diff --git a/docs/agent_handoffs/20261101_推免名单离线解析与删除边界.md b/docs/agent_handoffs/20261101_推免名单离线解析与删除边界.md new file mode 100644 index 000000000..1159de86e --- /dev/null +++ b/docs/agent_handoffs/20261101_推免名单离线解析与删除边界.md @@ -0,0 +1,41 @@ +# 2026-11-01 推免名单离线解析与原件删除边界 + +## 目标 + +在磁盘只剩 8.7G、agent 额度中断的约束下,不再新增下载,改吃已落盘的原件;把中断批次 `y2027_tuimian_round2` 解析入库,入库前后各核验一次,然后按用户指令在提交 Git 后删除原件。 + +## 本轮已完成 + +- 载体判定:`y2027_tuimian_round2/raw` 81 个文件里,63 个 HTML 多数只是入口页(东南大学、华东理工大学的名单页正文没有逐专业数值,名单挂在 PDF 附件),真正有数值的是 15 份官方公示 PDF 与合肥工业的 1679 行 HTML 名单表。 +- 三类载体分列处理,不合并: + - 官方逐专业计数表:`seu_stat`(普通 115 键 3051 人 + 强基 12 键 57 人)、`scut_plan`(统考可用计划 138 行 3322 人)。 + - 逐人名单:北师、华东师大、西北工业、南理工、东北大学、北中医、矿大(北京)、上海师大、合肥工业,按(学院, 学院代码, 专业代码, 专业名称, 层次, 学习方式)分组行数。 + - 无文字层扫描件:`scut_bs`(8 页)、`scut_ss`(60 页)缓存为空,判为需 OCR,本轮没有强行造数。 +- 新表 `y2027_tuimian_2027`(1436 行,11 校)与 `y2027_plan_2027`(138 行);未改动既有 `y2027_tuimian_major`,避免与并行会话相互覆盖。 +- 三处会导致假数的判据已修正: + - 汇总/合计行先排除再分组(东南计数表 34 行、强基表 8 行属院系级合计)。 + - 专项计划行不计入推免接收人数(东南 33 条研究生支教团、北中医 13 条),否则合计虚高。 + - PDF 抽出的 Kangxi 兼容部首(`⼈`/`⼯`/`⽣`)折回正规汉字,全角标点保持原样;这一改动使跨源关联键不再静默漏配,库内残留兼容部首计数为 0。 +- 学院列两种错法都处理了:`拟录取学院码及名称` 合并表头拆列后取到的是代码,先保代码再由内容定位补名称;院系列是合并单元格时向下填充。 +- 口径冲突留证:东南大学同一批数据既有官方计数表又有逐人名单,名单分组比官方计数多 278 人(差值随专项排除后仍在),判为口径差异(官方「录取数」对名单「拟录取公示」),不判谁错。按规则只入官方计数,182 行推导值丢弃不对账入库。 +- 校验器先被故意改坏验证过一轮:非整数、超上界、空专业键、同业务键重复四类各造一条,全部拒绝且探针行未落库;首轮暴露备份脚本取错 `description` 下标(列名在 `d[0]` 不是 `d[1]`),重做后回读核验行数与首行校名一致。 + +## 产物与证据 + +- 解析与入库脚本:`运行暂存/agent_ingest_20261031/` 下 `cache_pdf_grid.py`、`parse_round2_from_cache.py`、`finalize_round2.py`、`ingest_round2_roster.py`。 +- 候选与诊断:`cand_round2_pdf.json`、`cand_round2_final.json`、`cand_hfut.json`、`round2_pdf_diag.txt`、`seu_crosscheck.txt`、`postcheck_round2.txt`、`spotcheck_round2.txt`。 +- 表格缓存 `grid_cache/`(44 份 JSON,2.5M);原始 PDF 未改动。 +- 入库前备份 `运行暂存/入库前备份/20261101_tuimian_round2_pre/`:`y2027_tuimian_major.csv.gz`(1587 行,SHA-256 `f971a16b…`)+ 回读核验过的 `receipt.json`。 +- `dim_table_meta` 登记 2 表,`dim_field_semantics` 登记 9 条字段语义,`changelog` 2 条。 + +## 删除边界(提交 Git 后执行) + +- 可删:`运行暂存/y2027_tuimian_round2/raw` 中已解析入库的 15 份 PDF 与 `hfut_page_718c76.htm`;此前 A 级已入库并沉淀回执的批次(新东方、中公、启航、高顿、报录比、0856 目录、no_channel、score_lines 等,约 615M)。 +- 暂不可删:`运行暂存/official_roster_2025_2026/raw`(106M,29 份 PDF + 47 HTML 尚未解析入库);`运行暂存/scut_bs`、`scut_ss` 两份扫描件是唯一原件且未 OCR。 +- 不动:并行会话的 `审计报告/local_ingest_20261031/prewrite_snapshot/` 28G、`_归档/_待删_20261022` 4.9G、`ggyx_chrome/` 登录态。 + +## 未完成 + +1. `official_roster_2025_2026` 的 2025/2026 复试与拟录取名单还没解析(表格缓存已就绪),解析前不能删原件。 +2. 两份华南理工扫描件需 OCR 才有数值。 +3. 本轮新表未接入交付层,`y2027_tuimian_major` 与新表需在交付层按 basis 分列展示。 diff --git a/审计报告/推免名单解析_20261101/cand_round2_final.json.gz b/审计报告/推免名单解析_20261101/cand_round2_final.json.gz new file mode 100644 index 000000000..dff964b5a Binary files /dev/null and b/审计报告/推免名单解析_20261101/cand_round2_final.json.gz differ diff --git a/审计报告/推免名单解析_20261101/hfut_2027_tuimian.json b/审计报告/推免名单解析_20261101/hfut_2027_tuimian.json new file mode 100644 index 000000000..22cda7bca --- /dev/null +++ b/审计报告/推免名单解析_20261101/hfut_2027_tuimian.json @@ -0,0 +1,779 @@ +[ + { + "college": "机械工程学院", + "major_code": "085501", + "major_name": "机械工程", + "kind": "硕士", + "value": 113 + }, + { + "college": "电气与自动化工程学院", + "major_code": "085801", + "major_name": "电气工程", + "kind": "硕士", + "value": 102 + }, + { + "college": "管理与智能科学学院", + "major_code": "120100", + "major_name": "管理科学与工程", + "kind": "硕士", + "value": 87 + }, + { + "college": "电气与自动化工程学院", + "major_code": "080800", + "major_name": "电气工程", + "kind": "硕士", + "value": 55 + }, + { + "college": "计算机与信息学院", + "major_code": "081200", + "major_name": "计算机科学与技术", + "kind": "硕士", + "value": 49 + }, + { + "college": "计算机与信息学院", + "major_code": "085410", + "major_name": "人工智能", + "kind": "硕士", + "value": 42 + }, + { + "college": "数学学院", + "major_code": "070100", + "major_name": "数学", + "kind": "硕士", + "value": 41 + }, + { + "college": "仪器科学与光电工程学院", + "major_code": "085407", + "major_name": "仪器仪表工程", + "kind": "硕士", + "value": 39 + }, + { + "college": "食品与生物工程学院", + "major_code": "086003", + "major_name": "食品工程", + "kind": "硕士", + "value": 38 + }, + { + "college": "计算机与信息学院", + "major_code": "085404", + "major_name": "计算机技术", + "kind": "硕士", + "value": 36 + }, + { + "college": "微电子学院", + "major_code": "085401", + "major_name": "新一代电子信息技术(含量子技术等)", + "kind": "硕士", + "value": 36 + }, + { + "college": "管理与智能科学学院", + "major_code": "120200", + "major_name": "工商管理学", + "kind": "硕士", + "value": 35 + }, + { + "college": "食品与生物工程学院", + "major_code": "083200", + "major_name": "食品科学与工程", + "kind": "硕士", + "value": 35 + }, + { + "college": "经济学院", + "major_code": "025100", + "major_name": "金融", + "kind": "硕士", + "value": 34 + }, + { + "college": "材料科学与工程学院", + "major_code": "080500", + "major_name": "材料科学与工程", + "kind": "硕士", + "value": 33 + }, + { + "college": "材料科学与工程学院", + "major_code": "085601", + "major_name": "材料工程", + "kind": "硕士", + "value": 33 + }, + { + "college": "仪器科学与光电工程学院", + "major_code": "080400", + "major_name": "仪器科学与技术", + "kind": "硕士", + "value": 30 + }, + { + "college": "电气与自动化工程学院", + "major_code": "085406", + "major_name": "控制工程", + "kind": "硕士", + "value": 30 + }, + { + "college": "汽车与交通工程学院", + "major_code": "085502", + "major_name": "车辆工程", + "kind": "硕士", + "value": 27 + }, + { + "college": "计算机与信息学院", + "major_code": "085402", + "major_name": "通信工程(含宽带网络、移动通信等)", + "kind": "硕士", + "value": 26 + }, + { + "college": "土木与水利工程学院", + "major_code": "081400", + "major_name": "土木工程", + "kind": "硕士", + "value": 26 + }, + { + "college": "机械工程学院", + "major_code": "080201", + "major_name": "机械制造及其自动化", + "kind": "硕士", + "value": 22 + }, + { + "college": "机械工程学院", + "major_code": "080202", + "major_name": "机械电子工程", + "kind": "硕士", + "value": 22 + }, + { + "college": "机械工程学院", + "major_code": "080203", + "major_name": "机械设计及理论", + "kind": "硕士", + "value": 22 + }, + { + "college": "微电子学院", + "major_code": "080900", + "major_name": "电子科学与技术", + "kind": "硕士", + "value": 22 + }, + { + "college": "马克思主义学院", + "major_code": "030500", + "major_name": "马克思主义理论", + "kind": "硕士", + "value": 22 + }, + { + "college": "化学与化工学院", + "major_code": "081700", + "major_name": "化学工程与技术", + "kind": "硕士", + "value": 21 + }, + { + "college": "化学与化工学院", + "major_code": "085602", + "major_name": "化学工程", + "kind": "硕士", + "value": 19 + }, + { + "college": "土木与水利工程学院", + "major_code": "085901", + "major_name": "土木工程", + "kind": "硕士", + "value": 19 + }, + { + "college": "管理与智能科学学院", + "major_code": "120201", + "major_name": "会计学", + "kind": "硕士", + "value": 19 + }, + { + "college": "汽车与交通工程学院", + "major_code": "086100", + "major_name": "交通运输", + "kind": "硕士", + "value": 18 + }, + { + "college": "电气与自动化工程学院", + "major_code": "081100", + "major_name": "控制科学与工程", + "kind": "硕士", + "value": 17 + }, + { + "college": "计算机与信息学院", + "major_code": "081000", + "major_name": "信息与通信工程", + "kind": "硕士", + "value": 17 + }, + { + "college": "卓越工程师学院", + "major_code": "085502", + "major_name": "车辆工程", + "kind": "硕士", + "value": 17 + }, + { + "college": "经济学院", + "major_code": "020200", + "major_name": "应用经济学", + "kind": "硕士", + "value": 14 + }, + { + "college": "卓越工程师学院", + "major_code": "085501", + "major_name": "机械工程", + "kind": "硕士", + "value": 14 + }, + { + "college": "外国语学院", + "major_code": "055101", + "major_name": "英语笔译", + "kind": "硕士", + "value": 13 + }, + { + "college": "汽车与交通工程学院", + "major_code": "080204", + "major_name": "车辆工程", + "kind": "硕士", + "value": 13 + }, + { + "college": "卓越工程师学院", + "major_code": "085801", + "major_name": "电气工程", + "kind": "硕士", + "value": 13 + }, + { + "college": "仪器科学与光电工程学院", + "major_code": "083100", + "major_name": "生物医学工程", + "kind": "硕士", + "value": 12 + }, + { + "college": "化学与化工学院", + "major_code": "070300", + "major_name": "化学", + "kind": "硕士", + "value": 12 + }, + { + "college": "建筑与艺术学院", + "major_code": "135700", + "major_name": "设计", + "kind": "硕士", + "value": 12 + }, + { + "college": "食品与生物工程学院", + "major_code": "071000", + "major_name": "生物学", + "kind": "硕士", + "value": 12 + }, + { + "college": "汽车与交通工程学院", + "major_code": "085802", + "major_name": "动力工程", + "kind": "硕士", + "value": 12 + }, + { + "college": "集成电路创新学院", + "major_code": "085401", + "major_name": "新一代电子信息技术(含量子技术等)", + "kind": "硕士", + "value": 12 + }, + { + "college": "智能软件工程学院", + "major_code": "085405", + "major_name": "软件工程", + "kind": "硕士", + "value": 12 + }, + { + "college": "机械工程学院", + "major_code": "0802Z1", + "major_name": "工业工程", + "kind": "硕士", + "value": 11 + }, + { + "college": "土木与水利工程学院", + "major_code": "080100", + "major_name": "力学", + "kind": "硕士", + "value": 11 + }, + { + "college": "外国语学院", + "major_code": "050200", + "major_name": "外国语言文学", + "kind": "硕士", + "value": 11 + }, + { + "college": "汽车与交通工程学院", + "major_code": "082300", + "major_name": "交通运输工程", + "kind": "硕士", + "value": 10 + }, + { + "college": "经济学院", + "major_code": "020100", + "major_name": "理论经济学", + "kind": "硕士", + "value": 10 + }, + { + "college": "文法学院", + "major_code": "035102", + "major_name": "法律(法学)", + "kind": "硕士", + "value": 10 + }, + { + "college": "卓越工程师学院", + "major_code": "085601", + "major_name": "材料工程", + "kind": "硕士", + "value": 10 + }, + { + "college": "卓越工程师学院", + "major_code": "085500", + "major_name": "机械", + "kind": "直博生", + "value": 10 + }, + { + "college": "土木与水利工程学院", + "major_code": "085902", + "major_name": "水利工程", + "kind": "硕士", + "value": 8 + }, + { + "college": "土木与水利工程学院", + "major_code": "085905", + "major_name": "市政工程(含给排水等)", + "kind": "硕士", + "value": 8 + }, + { + "college": "资源与环境工程学院", + "major_code": "085701", + "major_name": "环境工程", + "kind": "硕士", + "value": 8 + }, + { + "college": "人工智能创新学院", + "major_code": "085410", + "major_name": "人工智能", + "kind": "硕士", + "value": 8 + }, + { + "college": "土木与水利工程学院", + "major_code": "081500", + "major_name": "水利工程", + "kind": "硕士", + "value": 7 + }, + { + "college": "资源与环境工程学院", + "major_code": "070500", + "major_name": "地理学", + "kind": "硕士", + "value": 7 + }, + { + "college": "资源与环境工程学院", + "major_code": "083000", + "major_name": "环境科学与工程", + "kind": "硕士", + "value": 7 + }, + { + "college": "食品与生物工程学院", + "major_code": "105500", + "major_name": "药学", + "kind": "硕士", + "value": 7 + }, + { + "college": "汽车与交通工程学院", + "major_code": "080700", + "major_name": "动力工程及工程热物理", + "kind": "硕士", + "value": 7 + }, + { + "college": "聚变科学与工程学院", + "major_code": "085402", + "major_name": "通信工程(含宽带网络、移动通信等)", + "kind": "硕士", + "value": 7 + }, + { + "college": "聚变科学与工程学院", + "major_code": "085801", + "major_name": "电气工程", + "kind": "硕士", + "value": 7 + }, + { + "college": "土木与水利工程学院", + "major_code": "081403", + "major_name": "市政工程", + "kind": "硕士", + "value": 6 + }, + { + "college": "建筑与艺术学院", + "major_code": "086200", + "major_name": "风景园林", + "kind": "硕士", + "value": 6 + }, + { + "college": "食品与生物工程学院", + "major_code": "0817Z1", + "major_name": "制药工程", + "kind": "硕士", + "value": 6 + }, + { + "college": "卓越工程师学院", + "major_code": "085410", + "major_name": "人工智能", + "kind": "硕士", + "value": 6 + }, + { + "college": "集成电路创新学院", + "major_code": "0812J1", + "major_name": "集成电路与系统", + "kind": "硕士", + "value": 6 + }, + { + "college": "土木与水利工程学院", + "major_code": "081404", + "major_name": "供热、供燃气、通风及空调工程", + "kind": "硕士", + "value": 5 + }, + { + "college": "建筑与艺术学院", + "major_code": "083300", + "major_name": "城乡规划学", + "kind": "硕士", + "value": 5 + }, + { + "college": "建筑与艺术学院", + "major_code": "085100", + "major_name": "建筑", + "kind": "硕士", + "value": 5 + }, + { + "college": "建筑与艺术学院", + "major_code": "085507", + "major_name": "工业设计工程", + "kind": "硕士", + "value": 5 + }, + { + "college": "建筑与艺术学院", + "major_code": "130100", + "major_name": "艺术学", + "kind": "硕士", + "value": 5 + }, + { + "college": "建筑与艺术学院", + "major_code": "140300", + "major_name": "设计学", + "kind": "硕士", + "value": 5 + }, + { + "college": "经济学院", + "major_code": "025400", + "major_name": "国际商务", + "kind": "硕士", + "value": 5 + }, + { + "college": "物理学院", + "major_code": "085401", + "major_name": "新一代电子信息技术(含量子技术等)", + "kind": "硕士", + "value": 5 + }, + { + "college": "卓越工程师学院", + "major_code": "085406", + "major_name": "控制工程", + "kind": "硕士", + "value": 5 + }, + { + "college": "人工智能创新学院", + "major_code": "081200", + "major_name": "计算机科学与技术", + "kind": "硕士", + "value": 5 + }, + { + "college": "聚变科学与工程学院", + "major_code": "085501", + "major_name": "机械工程", + "kind": "硕士", + "value": 5 + }, + { + "college": "智能软件工程学院", + "major_code": "081200", + "major_name": "计算机科学与技术", + "kind": "硕士", + "value": 5 + }, + { + "college": "资源与环境工程学院", + "major_code": "081800", + "major_name": "地质资源与地质工程", + "kind": "硕士", + "value": 4 + }, + { + "college": "食品与生物工程学院", + "major_code": "086001", + "major_name": "生物技术与工程", + "kind": "硕士", + "value": 4 + }, + { + "college": "食品与生物工程学院", + "major_code": "086002", + "major_name": "制药工程", + "kind": "硕士", + "value": 4 + }, + { + "college": "物理学院", + "major_code": "070200", + "major_name": "物理学", + "kind": "硕士", + "value": 4 + }, + { + "college": "文法学院", + "major_code": "030100", + "major_name": "法学", + "kind": "硕士", + "value": 4 + }, + { + "college": "聚变科学与工程学院", + "major_code": "070200", + "major_name": "物理学", + "kind": "硕士", + "value": 4 + }, + { + "college": "卓越工程师学院", + "major_code": "085800", + "major_name": "能源动力", + "kind": "直博生", + "value": 4 + }, + { + "college": "化学与化工学院", + "major_code": "080502", + "major_name": "材料学", + "kind": "硕士", + "value": 3 + }, + { + "college": "化学与化工学院", + "major_code": "085601", + "major_name": "材料工程", + "kind": "硕士", + "value": 3 + }, + { + "college": "卓越工程师学院", + "major_code": "085401", + "major_name": "新一代电子信息技术(含量子技术等)", + "kind": "硕士", + "value": 3 + }, + { + "college": "卓越工程师学院", + "major_code": "085404", + "major_name": "计算机技术", + "kind": "硕士", + "value": 3 + }, + { + "college": "聚变科学与工程学院", + "major_code": "081000", + "major_name": "信息与通信工程", + "kind": "硕士", + "value": 3 + }, + { + "college": "卓越工程师学院", + "major_code": "085400", + "major_name": "电子信息", + "kind": "直博生", + "value": 3 + }, + { + "college": "机械工程学院", + "major_code": "085802", + "major_name": "动力工程", + "kind": "硕士", + "value": 2 + }, + { + "college": "土木与水利工程学院", + "major_code": "081600", + "major_name": "测绘科学与技术", + "kind": "硕士", + "value": 2 + }, + { + "college": "土木与水利工程学院", + "major_code": "085704", + "major_name": "测绘工程", + "kind": "硕士", + "value": 2 + }, + { + "college": "土木与水利工程学院", + "major_code": "085906", + "major_name": "人工环境工程(含供热、通风及空调等)", + "kind": "硕士", + "value": 2 + }, + { + "college": "建筑与艺术学院", + "major_code": "081300", + "major_name": "建筑学", + "kind": "硕士", + "value": 2 + }, + { + "college": "聚变科学与工程学院", + "major_code": "085601", + "major_name": "材料工程", + "kind": "硕士", + "value": 2 + }, + { + "college": "电气与自动化工程学院", + "major_code": "085800", + "major_name": "能源动力", + "kind": "直博生", + "value": 2 + }, + { + "college": "管理与智能科学学院", + "major_code": "085400", + "major_name": "电子信息", + "kind": "直博生", + "value": 2 + }, + { + "college": "资源与环境工程学院", + "major_code": "085703", + "major_name": "地质工程", + "kind": "硕士", + "value": 1 + }, + { + "college": "物理学院", + "major_code": "080300", + "major_name": "光学工程", + "kind": "硕士", + "value": 1 + }, + { + "college": "卓越工程师学院", + "major_code": "085407", + "major_name": "仪器仪表工程", + "kind": "硕士", + "value": 1 + }, + { + "college": "材料科学与工程学院", + "major_code": "080500", + "major_name": "材料科学与工程", + "kind": "直博生", + "value": 1 + }, + { + "college": "电气与自动化工程学院", + "major_code": "085400", + "major_name": "电子信息", + "kind": "直博生", + "value": 1 + }, + { + "college": "资源与环境工程学院", + "major_code": "070900", + "major_name": "地质学", + "kind": "直博生", + "value": 1 + }, + { + "college": "管理与智能科学学院", + "major_code": "085500", + "major_name": "机械", + "kind": "直博生", + "value": 1 + }, + { + "college": "人工智能创新学院", + "major_code": "081200", + "major_name": "计算机科学与技术", + "kind": "直博生", + "value": 1 + } +] \ No newline at end of file diff --git a/审计报告/推免名单解析_20261101/log_cache_tm27.txt b/审计报告/推免名单解析_20261101/log_cache_tm27.txt new file mode 100644 index 000000000..7c3388082 --- /dev/null +++ b/审计报告/推免名单解析_20261101/log_cache_tm27.txt @@ -0,0 +1,16 @@ +Consider using the pymupdf_layout package for a greatly improved page layout analysis. +tm27_bnu_pdf_cbaa1326 pages=70 bytes=250812 +tm27_bucm_pdf2_0412bf pages=16 bytes=68023 +tm27_cumtb_pdf_0c26f3 pages=15 bytes=82605 +tm27_ecnu_pdf_ea6612af pages=50 bytes=232930 +tm27_neu_pdf_2293fa87 pages=116 bytes=361225 +tm27_njust_pdf_3fed60bb pages=80 bytes=175114 +tm27_nwpu_pdf_466e2335 pages=46 bytes=278807 +tm27_scut_bs_5698d74a pages=8 bytes=75 +tm27_scut_plan_38aada94 pages=3 bytes=8121 +tm27_scut_ss_f2e9d15f pages=60 bytes=283 +tm27_seu_qj_79a99de5 pages=3 bytes=10833 +tm27_seu_ss_c31b26d0 pages=89 bytes=446893 +tm27_seu_stat_8bab9f8e pages=3 bytes=10361 +tm27_seu_stat_qj_8a75a02e pages=1 bytes=1369 +tm27_shnu_pdf_2fce5d pages=5 bytes=18164 diff --git a/审计报告/推免名单解析_20261101/manifest.json b/审计报告/推免名单解析_20261101/manifest.json new file mode 100644 index 000000000..1e4cbf0ee --- /dev/null +++ b/审计报告/推免名单解析_20261101/manifest.json @@ -0,0 +1,35 @@ +{ + "generated_at": "2026-11-01", + "rows_in_cand": 1574, + "tables": { + "y2027_tuimian_2027": 1436, + "y2027_plan_2027": 138 + }, + "source_files_ingested": [ + "15份官方推免公示PDF", + "hfut_page_718c76.htm" + ], + "source_files_not_ingested": { + "scanned_no_text_layer": [ + "scut_bs_5698d74a.pdf", + "scut_ss_f2e9d15f.pdf" + ], + "entry_pages_without_table": "其余63个HTML多数是入口页,名单在PDF附件", + "dropped_after_crosscheck": { + "seu_ss_c31b26d0.pdf/seu_qj_79a99de5.pdf": "同校已有官方计数表,182行推导值只作对账" + } + }, + "backup_sha256": "f971a16b1a577e07e1a2e9a56e44706ed22dbf50a40997d2264868376c167a8c", + "files": { + "cand_round2_final.json.gz": "73b6f9d678f6091c83d0267c9ae6a42787aa0589c95598d05d322ffade3fc9b5", + "hfut_2027_tuimian.json": "748bee8f0e1818d9e47139ce35c855fd22bb2aa3ec260c0f35f4cdd391d747fb", + "log_cache_tm27.txt": "1665d9af4bcbf32d2c714c9dab6778cf1e2021c9c1917208c9d184ae31ef95d6", + "postcheck_round2.txt": "55c91e37e8e06a9e88cacf32dd190db70e8eb3055e5259e99b5bc76ee07ab0cf", + "receipt.json": "34b9a0412e021d04b72d5cf2fbd2062eabd393302e6c158e0421529bb5f0f840", + "round2_pdf_diag.txt": "12673e14b0c13b6923dcf2879701345d6595af2da158e1a5fb6e152d7deee438", + "round2_tag_index.json": "069bc2beb319c596e86913850f31235033138be93a1fa30dac5b010cf5f0222a", + "seu_crosscheck.txt": "057a29538585e580c659b3da9888e2834ff449e5b0262cf7b4ce1ea482cb86b9", + "spotcheck_round2.txt": "47644165d81b5e4c483e51d67fc1ba0462f66f6c98f3b279c5826f9980c3cc11", + "y2027_tuimian_major.csv.gz": "f971a16b1a577e07e1a2e9a56e44706ed22dbf50a40997d2264868376c167a8c" + } +} \ No newline at end of file diff --git a/审计报告/推免名单解析_20261101/postcheck_round2.txt b/审计报告/推免名单解析_20261101/postcheck_round2.txt new file mode 100644 index 000000000..1cba206f9 --- /dev/null +++ b/审计报告/推免名单解析_20261101/postcheck_round2.txt @@ -0,0 +1,31 @@ +=== 入库后核验 y2027_tuimian_2027 === + 上海师范大学 roster_derived 键=55 人数=271 区间=1~24 + 东北大学 roster_derived 键=99 人数=2307 区间=1~164 + 东南大学 count_direct 键=127 人数=3108 区间=1~190 + 中国矿业大学(北京) roster_derived 键=71 人数=624 区间=1~80 + 北京中医药大学 roster_derived 键=68 人数=405 区间=1~81 + 北京师范大学 roster_derived 键=225 人数=2268 区间=1~105 + 华东师范大学 roster_derived 键=365 人数=2429 区间=1~107 + 南京理工大学 roster_derived 键=128 人数=1832 区间=1~140 + 合肥工业大学 roster_derived 键=111 人数=1679 区间=1~113 + 西北工业大学 roster_derived 键=187 人数=2702 区间=1~124 + 总键=1436 总人数=17625 + 值域异常(<=0 或 >2000): 0 + 非数字值: 0 + 空专业键(code与name都空): 0 + 自设码(含字母): 24 + 残留CJK兼容部首: 0 + plan_type 分布: [('强基计划', 12), ('普通推免', 1424)] +=== y2027_plan_2027 === + ('华南理工大学', 138, 3322) + 空名称: 0 +=== 候选 vs 库内 键数对账 === + 上海师范大学 候选=55 库=55 OK + 东北大学 候选=99 库=99 OK + 东南大学 候选=127 库=127 OK + 中国矿业大学(北京) 候选=71 库=71 OK + 北京中医药大学 候选=68 库=68 OK + 北京师范大学 候选=225 库=225 OK + 华东师范大学 候选=365 库=365 OK + 南京理工大学 候选=128 库=128 OK + 西北工业大学 候选=187 库=187 OK diff --git a/审计报告/推免名单解析_20261101/receipt.json b/审计报告/推免名单解析_20261101/receipt.json new file mode 100644 index 000000000..35df22eff --- /dev/null +++ b/审计报告/推免名单解析_20261101/receipt.json @@ -0,0 +1,8 @@ +{ + "table": "y2027_tuimian_major", + "rows_before": 1587, + "sha256": "f971a16b1a577e07e1a2e9a56e44706ed22dbf50a40997d2264868376c167a8c", + "readback_rows": 1587, + "readback_first_school": "东华大学", + "at": "2026-11-01" +} \ No newline at end of file diff --git a/审计报告/推免名单解析_20261101/round2_pdf_diag.txt b/审计报告/推免名单解析_20261101/round2_pdf_diag.txt new file mode 100644 index 000000000..6dadabc65 --- /dev/null +++ b/审计报告/推免名单解析_20261101/round2_pdf_diag.txt @@ -0,0 +1,15 @@ +bnu 北京师范大学 hdr=1832 total_skip=0 nohm=1 行=49 直接计数=0 分组键=49 +bucm 北京中医药大学 hdr=204 total_skip=0 nohm=0 行=1 直接计数=0 分组键=1 +cumtb 中国矿业大学(北京) hdr=607 total_skip=0 nohm=1 行=0 直接计数=0 分组键=0 +ecnu 华东师范大学 hdr=1741 total_skip=0 nohm=1 行=83 直接计数=0 分组键=83 +neu 东北大学 hdr=2361 total_skip=0 nohm=0 行=5 直接计数=0 分组键=5 +njust 南京理工大学 hdr=1821 total_skip=0 nohm=0 行=2 直接计数=0 分组键=2 +nwpu 西北工业大学 hdr=2538 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0 +scut 华南理工大学 hdr=0 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0 +scut 华南理工大学 hdr=38 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0 +scut 华南理工大学 hdr=0 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0 +seu 东南大学 hdr=80 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0 +seu 东南大学 hdr=3222 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0 +seu 东南大学 hdr=101 total_skip=3 nohm=0 行=0 直接计数=0 分组键=0 +seu 东南大学 hdr=13 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0 +shnu 上海师范大学 hdr=5 total_skip=0 nohm=0 行=55 直接计数=0 分组键=55 \ No newline at end of file diff --git a/审计报告/推免名单解析_20261101/round2_tag_index.json b/审计报告/推免名单解析_20261101/round2_tag_index.json new file mode 100644 index 000000000..2b50ad837 --- /dev/null +++ b/审计报告/推免名单解析_20261101/round2_tag_index.json @@ -0,0 +1,322 @@ +{ + "scu_list": { + "school": "四川大学", + "url": "https://yz.scu.edu.cn/zsxx/newslist/a/gk" + }, + "bucm_page": { + "school": "北京中医药大学", + "url": "https://yanjiusheng.bucm.edu.cn/zsjy/sszs/28766041e1154bd998292bab68a0ce32.htm" + }, + "uir_home": { + "school": "国际关系学院", + "url": "https://yjszs.uir.cn/" + }, + "tju_201": { + "school": "天津大学-机械", + "url": "https://me.tju.edu.cn/info/1041/54111.htm" + }, + "tju_202": { + "school": "天津大学-精仪", + "url": "https://jyxy.tju.edu.cn/cn/new/20260928/2499.shtml" + }, + "tju_209": { + "school": "天津大学-经管", + "url": "https://come.tju.edu.cn/info/1013/18623.htm" + }, + "cumtb_pdf": { + "school": "中国矿业大学(北京)", + "url": "https://yz.cumtb.edu.cn/upload/uid1/b/0f/c3/98ac7032308e92d40608b2bd4dfa178d5cec30fb.pdf" + }, + "ncu_list": { + "school": "南昌大学", + "url": "http://yjsy.ncu.edu.cn/index/gszl.htm" + }, + "tyut_page": { + "school": "太原理工大学", + "url": "http://www.gs.tyut.edu.cn/info/1261/16789.htm" + }, + "ncu_page2": { + "school": "南昌大学", + "url": "http://yjsy.ncu.edu.cn/info/1014/47886.htm" + }, + "shnu_page": { + "school": "上海师范大学", + "url": "https://yjsc.shnu.edu.cn/1e/15/c17206a859669/page.htm" + }, + "hunnu_list": { + "school": "湖南师范大学", + "url": "https://yjsy.hunnu.edu.cn/zsks/sszk1/zkdt1.htm" + }, + "shnu_pdf": { + "school": "上海师范大学", + "url": "https://yjsc.shnu.edu.cn/_upload/article/files/7c/c0/0259cecb4b26b37cc4f095d20dbf/6251b6ea-1dec-4b7d-a400-9e4a00678a82.pdf" + }, + "hunnu_page2": { + "school": "湖南师范大学", + "url": "https://yjsy.hunnu.edu.cn/info/1028/16375.htm" + }, + "scu_sample": { + "school": "四川大学", + "url": "https://yz.scu.edu.cn/tms/tmsgs?yxs=101" + }, + "chsi_nju": { + "school": "南京大学", + "url": "https://yz.chsi.com.cn/sswbgg/pages/msgdetail.do?dwdm=10284&msg_id=574095" + }, + "cumt_list": { + "school": "中国矿业大学", + "url": "https://yz.cumt.edu.cn/zsxx/tzgg.htm" + }, + "hunnu_js": { + "school": "湖南师范大学", + "url": "https://yjsy.hunnu.edu.cn/zsks/sszk1/zkdt1/16375.htm" + }, + "bnu_page": { + "school": "北京师范大学", + "url": "https://yz.bnu.edu.cn/detail/2bdb9414-87e7-4509-b151-caba9e2ae4c5" + }, + "ecust_page": { + "school": "华东理工大学", + "url": "https://gschool.ecust.edu.cn/2026/1021/c12750a194415/page.htm" + }, + "scu_yxstmgs": { + "school": "四川大学", + "url": "https://yz.scu.edu.cn/tms/yxstmgs" + }, + "scu_jkgs": { + "school": "四川大学", + "url": "https://yz.scu.edu.cn/tms/tmsjkgs" + }, + "nju_home": { + "school": "南京大学", + "url": "https://yzb.nju.edu.cn/" + }, + "nju_news": { + "school": "南京大学", + "url": "https://yzb.nju.edu.cn/4794/list.htm" + }, + "nju_stat": { + "school": "南京大学", + "url": "https://yzb.nju.edu.cn/ea/44/c47863a846404/page.htm" + }, + "scu_yxs2": { + "school": "四川大学", + "url": "https://yz.scu.edu.cn/tms/yxstmgs" + }, + "neu_page": { + "school": "东北大学", + "url": "https://yz.neu.edu.cn/2026/1020/c5792a480951/page.htm" + }, + "dlut_page": { + "school": "大连理工", + "url": "https://gs.dlut.edu.cn/info/1173/52122.htm" + }, + "jlu_home": { + "school": "吉林大学", + "url": "https://yz.jlu.edu.cn/" + }, + "neu_page2": { + "school": "东北大学", + "url": "https://yz.neu.edu.cn/2026/1020/c5792a480951/page.htm" + }, + "dlut_page2": { + "school": "大连理工", + "url": "https://gs.dlut.edu.cn/info/1173/52122.htm" + }, + "nwpu_page": { + "school": "西北工业大学", + "url": "https://yzb.nwpu.edu.cn/info/1174/10568.htm" + }, + "njust_page": { + "school": "南京理工大学", + "url": "https://gs.njust.edu.cn/zsw/b3/aa/c4587a373674/page.htm" + }, + "bucm_pdf2": { + "school": "北京中医药大学", + "url": "https://yanjiusheng.bucm.edu.cn/docs/2026-10/33ea8b9a9ff045b08ec501ae8e5e74d1.pdf" + }, + "bjtu_page": { + "school": "北京交通大学", + "url": "https://yzb.bjtu.edu.cn/tzgg/134792e073ba4910b2f225d6068abd5d.htm" + }, + "ncepu_bj": { + "school": "华北电力大学", + "url": "https://yjsy.ncepu.edu.cn/zsxx/sszsxx/556a6dea89a8456aae0dbfada9811d11.htm" + }, + "ncepu_bd": { + "school": "华北电力大学(保定)", + "url": "https://gs.ncepu.edu.cn/zsxx/cxfw/7a74a16edf264830a215194d18436229.htm" + }, + "zzu_home": { + "school": "郑州大学", + "url": "https://yz.zzu.edu.cn/" + }, + "zzu_page2": { + "school": "郑州大学", + "url": "https://yz.zzu.edu.cn/info/1070/2436.htm" + }, + "cumt_page": { + "school": "中国矿业大学", + "url": "https://yz.cumt.edu.cn/info/1004/2615.htm" + }, + "cumtb_list": { + "school": "中国矿业大学(北京)", + "url": "https://yz.cumtb.edu.cn/index/newslist.html?type=1&class_id=156" + }, + "bjtu_home": { + "school": "北京交通大学", + "url": "https://yzb.bjtu.edu.cn/" + }, + "zzu_page3": { + "school": "郑州大学", + "url": "https://yz.zzu.edu.cn/info/1070/2436.htm" + }, + "cumt_page2": { + "school": "中国矿业大学", + "url": "https://yz.cumt.edu.cn/info/1004/2615.htm" + }, + "cumtb_list2": { + "school": "中国矿业大学(北京)", + "url": "https://yz.cumtb.edu.cn/index/newslist.html?type=1&class_id=156" + }, + "hfut_page": { + "school": "合肥工业大学", + "url": "https://yjszs.hfut.edu.cn/2026/1015/c13533a323798/pagem.htm" + }, + "fzu_list": { + "school": "福州大学", + "url": "https://yjsy.fzu.edu.cn/index/gsl/gsl.htm" + }, + "zzu_pdf": { + "school": "郑州大学", + "url": "https://yz.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1876580606&wbfileid=18210509" + }, + "cumtb_page": { + "school": "中国矿业大学(北京)", + "url": "https://yz.cumtb.edu.cn/index/detail.html?id=06ad8b9a960651" + }, + "fzu_page": { + "school": "福州大学", + "url": "https://yjsy.fzu.edu.cn/info/1077/6211.htm" + }, + "bnu_pdf": { + "school": "北京师范大学", + "url": "https://yz.bnu.edu.cn/content/docs/ss_nlqmd/2027%E5%85%AC%E7%A4%BA%E6%8E%A8%E5%85%8D%E6%8B%9F%E5%BD%95%E5%8F%96%E5%90%8D%E5%8D%95.pdf" + }, + "ecnu_pdf": { + "school": "华东师大", + "url": "https://yjszs.ecnu.edu.cn/_upload/article/files/fd/42/0b3b95dd435091470426d3262526/02854f30-1e4a-4ad3-b38a-80b5afa5c1b9.pdf" + }, + "ncu_page": { + "school": "南昌大学", + "url": "http://yjsy.ncu.edu.cn/info/1014/47886.htm" + }, + "neu_pdf": { + "school": "东北大学", + "url": "https://yz.neu.edu.cn/_upload/article/files/31/d8/12345b4c4433b7c87f1761a8e1d3/e23c36d2-64ad-4d3c-a765-b57c30276adc.pdf" + }, + "dlut_pdf": { + "school": "大连理工", + "url": "https://gs.dlut.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1502106601&wbfileid=1C922D0828D21F37D177FBD370C8E2C0" + }, + "nwpu_pdf": { + "school": "西北工业大学", + "url": "https://yzb.nwpu.edu.cn/__local/6/88/AA/EA4062A7A3EE1A67139DC748D75_4F482E24_ABA6D.pdf" + }, + "njust_pdf": { + "school": "南京理工大学", + "url": "https://gs.njust.edu.cn/_upload/article/files/d3/20/f77f1f2b46a18a938b187549c9bc/be959d85-dd4f-4794-9ec4-336ff2b32000.pdf" + }, + "ecnu": { + "school": "华东师大", + "url": "https://yjszs.ecnu.edu.cn/tzgg_43463/list.htm" + }, + "csu": { + "school": "中南", + "url": "https://yz.csu.edu.cn/" + }, + "xmu": { + "school": "厦门大学", + "url": "https://zs.xmu.edu.cn/sss.htm" + }, + "hnu": { + "school": "湖南大学", + "url": "https://gra.hnu.edu.cn/index/zsxx.htm" + }, + "bit": { + "school": "北京理工", + "url": "https://yzb.bit.edu.cn/" + }, + "nuaa": { + "school": "南航", + "url": "https://yzb.nuaa.edu.cn/" + }, + "tongji": { + "school": "同济", + "url": "https://yz.tongji.edu.cn/zsxw/ggtz.htm" + }, + "uestc": { + "school": "电子科大", + "url": "https://yz.uestc.edu.cn/sszs/tms.htm" + }, + "hust": { + "school": "华科", + "url": "https://gszs.hust.edu.cn/zsgg/sszs.htm" + }, + "fudan2": { + "school": "复旦", + "url": "https://gsao.fudan.edu.cn/15020/list2.htm" + }, + "ecnu_page": { + "school": "华东师大", + "url": "https://yjszs.ecnu.edu.cn/0c/6d/c43463a789613/page.htm" + }, + "nuaa_page": { + "school": "南京航空航天大学", + "url": "https://www.graduate.nuaa.edu.cn/2026/1012/c13502a414577/page.htm" + }, + "xmu_tms": { + "school": "厦门大学", + "url": "https://zs.xmu.edu.cn/sss/zsjz/tms.htm" + }, + "csu_tm": { + "school": "中南", + "url": "https://yz.csu.edu.cn/tms.htm" + }, + "hnu_list": { + "school": "湖南大学", + "url": "https://gra.hnu.edu.cn/index/zsxx.htm" + }, + "scut_ss": { + "school": "华南理工大学", + "url": "https://yz.scut.edu.cn/_upload/article/files/3e/8a/b77f81504aed9a0e4c99014a6f71/9a0b2657-7f75-4b61-a3d2-e010c414bbf4.pdf" + }, + "scut_bs": { + "school": "华南理工大学", + "url": "https://yz.scut.edu.cn/_upload/article/files/3e/8a/b77f81504aed9a0e4c99014a6f71/523dc474-d4f7-4966-8faf-45c6be6feca4.pdf" + }, + "scut_plan": { + "school": "华南理工大学", + "url": "https://yz.scut.edu.cn/_upload/article/files/3e/8a/b77f81504aed9a0e4c99014a6f71/e89c2239-c5ac-463d-b3fb-94b95fa53492.pdf" + }, + "seu_ss": { + "school": "东南大学", + "url": "https://yzb.seu.edu.cn/_upload/article/files/31/38/fa8cc19649cfa91ba1dd178b4807/0377af62-fc21-4647-bdb8-0aeb862ddd00.pdf" + }, + "seu_qj": { + "school": "东南大学", + "url": "https://yzb.seu.edu.cn/_upload/article/files/31/38/fa8cc19649cfa91ba1dd178b4807/352760b3-26f4-41bc-a4f4-ae04a8f1c082.pdf" + }, + "seu_stat": { + "school": "东南大学", + "url": "https://yzb.seu.edu.cn/_upload/article/files/31/38/fa8cc19649cfa91ba1dd178b4807/e2fa7c8b-328f-472d-bca9-1b55a39a41f8.pdf" + }, + "seu_stat_qj": { + "school": "东南大学", + "url": "https://yzb.seu.edu.cn/_upload/article/files/31/38/fa8cc19649cfa91ba1dd178b4807/1fe20865-27b2-4b86-b27b-596a5d7dceff.pdf" + }, + "zzu_page": { + "school": "郑州大学", + "url": "https://yz.zzu.edu.cn/info/1070/2436.htm" + } +} \ No newline at end of file diff --git a/审计报告/推免名单解析_20261101/seu_crosscheck.txt b/审计报告/推免名单解析_20261101/seu_crosscheck.txt new file mode 100644 index 000000000..68779bdc1 --- /dev/null +++ b/审计报告/推免名单解析_20261101/seu_crosscheck.txt @@ -0,0 +1,8 @@ +排除专项后:官方计数=3051人 名单分组=3329人 +对齐键=115 完全相等=63 占比=54.8% +剩余差异键=52 差值合计=270 + (('004', '081000'), 124, 179) + (('404', '085403'), 132, 165) + (('010', '070200'), 22, 39) + (('004', '080904'), 82, 98) + (('004', '085400'), 142, 157) diff --git a/审计报告/推免名单解析_20261101/spotcheck_round2.txt b/审计报告/推免名单解析_20261101/spotcheck_round2.txt new file mode 100644 index 000000000..1fa40f51a --- /dev/null +++ b/审计报告/推免名单解析_20261101/spotcheck_round2.txt @@ -0,0 +1,14 @@ +--- 抽查 5 行 --- + ('东南大学', '能源与环境学院', '089901', '碳中和科学技术', '', '4', 'count_direct') + ('北京师范大学', '物理与天文学院', '', '天文学', '博士', '1', 'roster_derived') + ('北京师范大学', '物理与天文学院', '', '物理教育', '硕士', '3', 'roster_derived') + ('中国矿业大学(北京)', '能源与矿业学院', '0819J4', '储能科学与工程', '硕士', '2', 'roster_derived') + ('北京师范大学', '未来教育学院', '', '学科教学(数学)', '硕士', '18', 'roster_derived') + +--- 原文比对:西北工业大学 080100 力学 --- + 缓存内 080100 名单行数=81 库内值=('5',) +--- 原文比对:合肥工业 085501 --- + 候选=[{'college': '机械工程学院', 'major_code': '085501', 'major_name': '机械工程', 'kind': '硕士', 'value': 113}, {'college': '卓越工程师学院', 'major_code': '085501', 'major_name': '机械工程', 'kind': '硕士', 'value': 14}, {'college': '聚变科学与工程学院', 'major_code': '085501', 'major_name': '机械工程', 'kind': '硕士', 'value': 5}] 库内=[('机械与电气工程学院', '42'), ('机械工程与自动化学院', '136'), ('机械工程与自动化学院', '2'), ('机械工程学院', '92'), ('中法工程师学院', '12'), ('机电学院', '27'), ('机械工程学院', '113'), ('卓越工程师学院', '14'), ('聚变科学与工程学院', '5')] +--- 东南大学强基与常规分列 --- + ('强基计划', 12, 57) + ('普通推免', 115, 3051) diff --git a/审计报告/推免名单解析_20261101/y2027_tuimian_major.csv.gz b/审计报告/推免名单解析_20261101/y2027_tuimian_major.csv.gz new file mode 100644 index 000000000..ff8d59d56 Binary files /dev/null and b/审计报告/推免名单解析_20261101/y2027_tuimian_major.csv.gz differ diff --git a/脚本/audit_all_tables.py b/脚本/audit_all_tables.py index 03bc460a1..62fdebc27 100644 --- a/脚本/audit_all_tables.py +++ b/脚本/audit_all_tables.py @@ -140,7 +140,7 @@ for idx, t in enumerate(tabs, 1): except Exception: pass profile.append([t, c, typ or '', n, nn, miss, nd, mn, mx]) - if miss > 50 and nn > 0 and t not in SEM_WILD: + if miss > 50 and nn > 0 and t not in SEM_WILD and (t, c) not in SEM: add('P4', '高缺失字段', c, '空值率 %.1f%%' % miss, n - nn) if is_num and nn: FLAGGED = "IFNULL(data_flag,'')<>''" if 'data_flag' in cn else '0' diff --git a/脚本/名单解析/cache_pdf_grid.py b/脚本/名单解析/cache_pdf_grid.py new file mode 100644 index 000000000..e30afddd3 --- /dev/null +++ b/脚本/名单解析/cache_pdf_grid.py @@ -0,0 +1,55 @@ +# -*- coding: utf-8 -*- +"""一次性把名单 PDF 的表格 grid 缓存成 JSON,后续解析迭代只读缓存。 + +find_tables 在百页级名单上极慢,重复试错等于每次重跑十几分钟,故先缓存。 +缓存只含表格单元格的文本,不含图片,体积远小于原件。 +""" +import glob +import json +import os +import sys + +import fitz + +OUTDIR = r'D:/择校/运行暂存/agent_ingest_20261031/grid_cache' +SRC = sys.argv[1] if len(sys.argv) > 1 else r'D:/择校/运行暂存/y2027_tuimian_round2/raw' +TAG = sys.argv[2] if len(sys.argv) > 2 else 'tm27' # 输出文件名前缀,两批目录可能同名 + + +def main(): + os.makedirs(OUTDIR, exist_ok=True) + for p in sorted(glob.glob(SRC + '/*.pdf')): + stem = TAG + '_' + os.path.basename(p).rsplit('.', 1)[0] + dst = os.path.join(OUTDIR, stem + '.json') + if os.path.exists(dst): + print('skip', stem) + continue + try: + doc = fitz.open(p) + except Exception as e: + print('FAIL', stem, e) + continue + pages = [] + for pi, page in enumerate(doc): + try: + tabs = list(page.find_tables().tables) + except Exception: + tabs = [] + grid = [] + for tb in tabs: + try: + for row in tb.extract(): + grid.append([('' if c is None else str(c)) for c in row]) + except Exception: + continue + if not grid: # 无结构表格时退到纯文本行 + grid = [[l] for l in page.get_text().split('\n') if l.strip()] + pages.append(grid) + doc.close() + with open(dst, 'w', encoding='utf-8') as f: + json.dump({'file': os.path.basename(p), 'pages': pages}, f, ensure_ascii=False) + print('%-26s pages=%d bytes=%d' % (stem, len(pages), os.path.getsize(dst))) + + +if __name__ == '__main__': + main() diff --git a/脚本/名单解析/finalize_round2.py b/脚本/名单解析/finalize_round2.py new file mode 100644 index 000000000..fdd7284d8 --- /dev/null +++ b/脚本/名单解析/finalize_round2.py @@ -0,0 +1,59 @@ +# -*- coding: utf-8 -*- +"""合并本轮名单解析候选,按口径规则过滤后入库。 + +三条过滤规则各有出处: + 1. 同一校已有官方逐专业计数表时,名单分组推导值只作对账证据,不入库(东南)。 + 2. 计划表里落不进计数列的残行不是推免人数,丢弃而不是标 flag(华南理工 2 行)。 + 3. 专项计划行(支教团等)已在解析阶段排除,此处不重复计数。 +""" +import json + +DROP_DERIVED = {'seu_ss_c31b26d0.pdf', 'seu_qj_79a99de5.pdf'} +DROP_PLAN_ROWS = {'scut_plan_38aada94.pdf'} + + +def main(): + pdf = json.load(open(r'D:/择校/运行暂存/agent_ingest_20261031/cand_round2_pdf.json', + encoding='utf-8')) + hfut = json.load(open(r'D:/择校/运行暂存/agent_ingest_20261031/cand_hfut.json', + encoding='utf-8')) + for x in hfut: + x.setdefault('college_code', '') + rows = pdf + hfut + kept, dropped = [], {'seu_derived': 0, 'plan_badrow': 0} + for x in rows: + if x['local_file'] in DROP_DERIVED and x.get('basis') == 'roster_derived': + dropped['seu_derived'] += 1 + continue + if x['local_file'] in DROP_PLAN_ROWS and x.get('basis') == 'roster_derived': + dropped['plan_badrow'] += 1 + continue + note = [] + if x['basis'] == 'count_direct': + note.append('官方逐专业计数表' if x.get('target') != 'plan' else '官方统考可用计划表') + else: + note.append('名单分组推导,已排除专项计划行') + if not x.get('major_code'): + note.append('原表无专业代码') + if not x.get('major_name'): + note.append('原表无专业名称') + if x['local_file'].startswith('seu_qj'): + note.append('强基计划单独表') + if x['local_file'].startswith('seu_qj') or x['local_file'].startswith('seu_stat_qj'): + x['plan_type'] = '强基计划' + elif x.get('target') == 'plan': + x['plan_type'] = '统考可用计划' + else: + x['plan_type'] = '普通推免' + x['note'] = ';'.join(note) + x['value'] = str(x['value']) + x.pop('_count_label', None) + kept.append(x) + dst = r'D:/择校/运行暂存/agent_ingest_20261031/cand_round2_final.json' + with open(dst, 'w', encoding='utf-8') as f: + json.dump(kept, f, ensure_ascii=False, indent=1) + print('保留 %d 丢弃 %s' % (len(kept), dropped)) + + +if __name__ == '__main__': + main() diff --git a/脚本/名单解析/ingest_round2_roster.py b/脚本/名单解析/ingest_round2_roster.py new file mode 100644 index 000000000..149f915bd --- /dev/null +++ b/脚本/名单解析/ingest_round2_roster.py @@ -0,0 +1,123 @@ +# -*- coding: utf-8 -*- +"""2027 推免/计划离线解析结果入库(新表,不动既有 y2027_tuimian_major)。 + +分列口径: + y2027_tuimian_2027.basis = count_direct 原表自带「录取数」列,官方实测 + y2027_tuimian_2027.basis = roster_derived 逐人名单分组行数,推导值 + y2027_plan_2027 「统考可用计划」,与推免人数不同口径,单表存 +入库前核验:代码格式、值为正整数且在上界内、空键拒绝、业务键去重。 +入库后核验:行数、按校分布、值域、与源文件行数对账。 +""" +import json +import os +import re +import sqlite3 +import sys +import time + +ZX = r'D:/择校/数据库/zexiao_2027.db' +CODE_OK = re.compile(r'^\d{6}$|^\d{4}[A-Z]\d{1,2}$|^\d{2}[A-Z]\d{3,4}$') # 含字母自设码合法 +MAX_COUNT = 3000 + +DDL_TM = '''CREATE TABLE IF NOT EXISTS y2027_tuimian_2027 ( + "school" TEXT, "year" TEXT, "college" TEXT, "college_code" TEXT, + "major_code" TEXT, "major_name" TEXT, "kind" TEXT, + "value" TEXT, "basis" TEXT, "plan_type" TEXT, "source_url" TEXT, "local_file" TEXT, + "note" TEXT, "data_flag" TEXT, "_ingested_at" TEXT, + PRIMARY KEY ("school","year","college","college_code","major_code","major_name", + "kind","basis","plan_type") +)''' +DDL_PLAN = '''CREATE TABLE IF NOT EXISTS y2027_plan_2027 ( + "school" TEXT, "year" TEXT, "college" TEXT, "college_code" TEXT, + "major_code" TEXT, "major_name" TEXT, "study_mode" TEXT, + "value" TEXT, "kind" TEXT, "source_url" TEXT, "local_file" TEXT, + "note" TEXT, "data_flag" TEXT, "_ingested_at" TEXT, + PRIMARY KEY ("school","year","college","college_code","major_code","major_name","study_mode","kind") +)''' + + +def validate(rows, keys): + """入库前核验,返回 (通过行, 拒绝原因计数)。""" + bad = {} + seen, out = set(), [] + for r in rows: + v = str(r.get('value', '')).strip() + if not v.isdigit(): + bad['value_not_int'] = bad.get('value_not_int', 0) + 1 + continue + n = int(v) + if n <= 0 or n > MAX_COUNT: + bad['value_out_of_range'] = bad.get('value_out_of_range', 0) + 1 + continue + code = (r.get('major_code') or '').strip() + if code and not CODE_OK.match(code): + bad['code_format'] = bad.get('code_format', 0) + 1 + r = dict(r, data_flag='code_format_odd') + if not code and not (r.get('major_name') or '').strip(): + bad['empty_major_key'] = bad.get('empty_major_key', 0) + 1 + continue + key = tuple((r.get(k) or '') for k in keys) + if key in seen: + bad['dup_business_key'] = bad.get('dup_business_key', 0) + 1 + continue + seen.add(key) + out.append(r) + return out, bad + + +def ingest(rows, table, tag, reject): + z = sqlite3.connect(ZX, timeout=300) + z.execute(DDL_TM if table == 'y2027_tuimian_2027' else DDL_PLAN) + cols = [d[0] for d in z.execute('select * from "%s" limit 0' % table).description + if d[0] != '_ingested_at'] + now = time.strftime('%Y-%m-%dT%H:%M:%S') + sql = 'INSERT OR IGNORE INTO "%s" ("%s","_ingested_at") VALUES (%s,?)' % ( + table, '","'.join(cols), ','.join(['?'] * len(cols))) + n = 0 + for r in rows: + rec = [] + for c in cols: + v = r.get(c, '') + if v is None: + v = '' + rec.append(str(v) if not isinstance(v, str) else v) + cur = z.execute(sql, rec + [now]) + n += cur.rowcount + z.commit() + z.execute('INSERT INTO changelog ("ts","script","object","action","n_rows","detail") ' + 'VALUES (?,?,?,?,?,?)', + (time.strftime('%Y-%m-%d %H:%M:%S'), 'round2_roster_20261101', table, + 'insert', n, + 'basis=%s cand=%d reject=%s' % (tag, len(rows), + json.dumps(reject, ensure_ascii=False)))) + z.commit() + tot = z.execute('select count(*) from "%s"' % table).fetchone()[0] + print('%s: 新增 %d / 候选 %d 表内合计 %d 拒绝 %s' % (table, n, len(rows), tot, reject)) + return z, n, tot + + +if __name__ == '__main__': + src = sys.argv[1] + data = json.load(open(src, encoding='utf-8')) + tm = [x for x in data if x.get('target') == 'tuimian'] + pl = [x for x in data if x.get('target') == 'plan'] + K_TM = ('school', 'year', 'college', 'college_code', 'major_code', 'major_name', + 'kind', 'basis', 'plan_type') + K_PL = ('school', 'year', 'college', 'college_code', 'major_code', 'major_name', + 'study_mode', 'kind') + tm, rj1 = validate(tm, K_TM) + pl, rj2 = validate(pl, K_PL) + print('候选 tuimian=%d plan=%d' % (len(tm), len(pl))) + if rj1: + print(' tuimian 拒绝:', rj1) + if rj2: + print(' plan 拒绝:', rj2) + z = None + if tm: + z, n1, t1 = ingest(tm, 'y2027_tuimian_2027', 'tuimian', rj1) + if pl: + zz, n2, t2 = ingest(pl, 'y2027_plan_2027', 'plan', rj2) + if z is None: + z = zz + if z: + z.close() diff --git a/脚本/名单解析/parse_round2_from_cache.py b/脚本/名单解析/parse_round2_from_cache.py new file mode 100644 index 000000000..53fcf5278 --- /dev/null +++ b/脚本/名单解析/parse_round2_from_cache.py @@ -0,0 +1,265 @@ +# -*- coding: utf-8 -*- +"""从 grid 缓存解析名单:严格表头判定 + 内容定位兜底。 + +上一版把每行都判成表头(数据行的学院值本身含「学院」二字),导致整表吞光。 +现在: + 1. 表头必须出现「姓名/考生编号/证件号码/身份证号/录取数/可用计划」这类只可能在表头出现的词, + 且该行不含 6 位专业代码。 + 2. 表头能定列就按表头;定不到(如东北大学「代码及名称」合并表头、实际拆成两列)就按内容定位: + 6 位代码左侧最近的中文短格=学院,右侧最近的中文=专业名。 + 3. 院系列是合并单元格,空值继承上一行。 + 4. 「汇总/合计/小计」行先排除再分组。 +""" +import glob +import json +import os +import re +import sys + +CACHE = r'D:/择校/运行暂存/agent_ingest_20261031/grid_cache' +SCHOOL = {'bnu': '北京师范大学', 'bucm': '北京中医药大学', 'cumtb': '中国矿业大学(北京)', + 'ecnu': '华东师范大学', 'neu': '东北大学', 'njust': '南京理工大学', + 'nwpu': '西北工业大学', 'scut': '华南理工大学', 'seu': '东南大学', + 'shnu': '上海师范大学'} +SCHOOL_ALIAS = {'华东师大': '华东师范大学'} +TOTAL_WORDS = ('汇总', '合计', '小计', '共计', '总计') +HDR_WORDS = re.compile(r'姓名|考生编号|证件号码|身份证号|录取数|可用计划|接收人数|拟接收|综合考核|复试成绩') +CODE_RE = re.compile(r'^\d{6}$|^\d{4}[A-Z]\d{1,2}$|^\d{2}[A-Z]\d{3,4}$') +ANYCODE = re.compile(r'(? best: + best, url = len(t), v['url'] + hm, rows = None, [] + last_college = '' + count_label = '' + n_hdr = n_skip_total = n_body = n_content = n_special = 0 + for pages in d['pages']: + for raw in pages: + cells = [clean(c) for c in raw] + if not any(cells): + continue + if is_header(cells): + cand = header_map(cells) + if cand: + hm, n_hdr = cand, n_hdr + 1 + count_label = (cells[cand['count']] if 'count' in cand + and cand['count'] < len(cells) else '') + last_college = '' + continue + if any(w in ''.join(cells) for w in TOTAL_WORDS): + n_skip_total += 1 + continue + code = name = college = kind = mode = count = college_code = '' + special = '' + if hm: + def at(k): + i = hm.get(k) + return cells[i] if (i is not None and i < len(cells)) else '' + college_code = at('college_code') + special = at('special') + v = at('code') + m = ANYCODE.search(v.replace(' ', '')) if v else None + code = m.group(1) if m else '' + name = at('name') + if m and not name and code and len(v.replace(' ', '')) > len(code): + tail = v.replace(' ', '').split(code, 1)[-1] # 「080200机械工程」连写 + name = re.sub(r'^[一-龥()()·A-Za-z0-9]{0,4}?(?=[一-龥])', '', tail)[:40] if re.search(r'[一-龥]', tail) else '' + name = tail if re.search(r'[一-龥]', tail) else '' + kind = at('kind') + mode = at('mode') + count = at('count') + college = at('college') or last_college or college_code + if code or name: + if (not name) or (not college) or college.isdigit(): + lc = locate_by_content(cells) # 「码及名称」合并表头会拆列 + if lc: + if college.isdigit() and len(college) <= 4: + college_code = college_code or college + if not college or college.isdigit(): + college = lc[0] or college + name = name or lc[1] + n_content += 1 + elif not code and not name: + lc = locate_by_content(cells) + if lc: + college = college or lc[0] + name = lc[1] + n_content += 1 + if not code: + for v in cells: + m = ANYCODE.search(v.replace(' ', '')) + if m and v.replace(' ', '') != name: + code = m.group(1) + break + else: + lc = locate_by_content(cells) + if not lc: + continue + college, name = lc[0] or last_college, lc[1] + for v in cells: + m = ANYCODE.search(v.replace(' ', '')) + if m: + code = m.group(1) + break + n_content += 1 + if special and not re.search(r'^0$|普通计划|无|^-$', special.replace(' ', '')): + n_special += 1 + continue # 专项计划不计入推免接收人数,否则合计虚高 + if college.isdigit() and len(college) <= 4: + college_code = college_code or college + college = '' # 代码列被当成名称列:搬回 college_code + if college: + last_college = college + if not (code or name): + continue + rows.append(dict(school=school, year=year, college=college, + college_code=college_code, major_code=code, + major_name=name, kind=kind, study_mode=mode, value=count, + source_url=url, local_file=d['file'], + _count_label=count_label)) + n_body += 1 + diag = '%-10s %-16s hdr=%-3d 总行=%-5d 汇总行=%-3d 专项行=%-4d 内容兜底=%-4d 无表头=%s' % ( + key, school, n_hdr, n_body, n_skip_total, n_special, n_content, + 'yes' if hm is None else 'no') + return school, rows, diag + + +def group(school, rows): + """直接计数行原样保留;名单行按业务键分组。""" + direct, agg, order = [], {}, [] + for r in rows: + if r['value'] and re.match(r'^\d{1,5}$', r['value']): + rr = dict(r, value=r['value'], basis='count_direct') + rr['target'] = 'plan' if '计划' in (r.get('_count_label') or '') else 'tuimian' + direct.append(rr) + continue + k = (r['school'], r['year'], r['college'], r['college_code'], r['major_code'], + r['major_name'], r['kind'], r['study_mode']) + if k not in agg: + agg[k] = dict(r, value='1', basis='roster_derived') + order.append(k) + else: + agg[k]['value'] = str(int(agg[k]['value']) + 1) + for r in agg.values(): + r['target'] = 'tuimian' + return direct, [agg[k] for k in order] + + +def main(): + tag = sys.argv[1] if len(sys.argv) > 1 else 'tm27' + year = '2027' + url_index = json.load(open(r'D:/择校/运行暂存/agent_ingest_20261031/round2_tag_index.json', + encoding='utf-8')) + out, diag = [], [] + for p in sorted(glob.glob(os.path.join(CACHE, tag + '_*.json'))): + try: + school, rows, d = parse(p, year, url_index) + except Exception as e: + diag.append('%s FAIL %s' % (os.path.basename(p), e)) + continue + direct, derived = group(school, rows) + diag.append(d + ' 直接计数=%d 分组=%d' % (len(direct), len(derived))) + out += direct + derived + dst = r'D:/择校/运行暂存/agent_ingest_20261031/cand_round2_pdf.json' + with open(dst, 'w', encoding='utf-8') as f: + json.dump(out, f, ensure_ascii=False, indent=1) + print('\n'.join(diag)) + print('rows=%d -> %s' % (len(out), dst)) + + +if __name__ == '__main__': + main()