沉淀2027推免名单离线解析与原件删除边界

磁盘告急且采集额度中断,改为只吃已落盘原件:15份官方推免公示PDF与
合肥工业1679行名单解析成 y2027_tuimian_2027(11校1436行),华南理工
统考可用计划单独入 y2027_plan_2027,避免与推免接收人数混口径。

判据修正三处会造出假数的地方:汇总/合计行先排除再分组、研究生支教团等
专项计划行不计入接收人数、PDF抽出的Kangxi兼容部首折回正规汉字(否则
跨源关联键静默漏配)。东南大学同批既有官方计数表又有逐人名单,名单分组
多278人判为口径差异,只入官方计数、182行推导值留作对账不入库。

校验器先被故意改坏验证过一轮才用于正式入库;备份脚本取错 description
下标导致备份写成空值,已重做并回读核验。
This commit is contained in:
Your Name committed 2026-11-01 00:25:29 +08:00
1 parent 82b461b127
commit 7d596c867e
17 files changed
+1772 -1

No files matched your search

@@ -0,0 +1,41 @@
# 2026-11-01 推免名单离线解析与原件删除边界
## 目标
在磁盘只剩 8.7G、agent 额度中断的约束下,不再新增下载,改吃已落盘的原件;把中断批次 `y2027_tuimian_round2` 解析入库,入库前后各核验一次,然后按用户指令在提交 Git 后删除原件。
## 本轮已完成
- 载体判定:`y2027_tuimian_round2/raw` 81 个文件里,63 个 HTML 多数只是入口页(东南大学、华东理工大学的名单页正文没有逐专业数值,名单挂在 PDF 附件),真正有数值的是 15 份官方公示 PDF 与合肥工业的 1679 行 HTML 名单表。
- 三类载体分列处理,不合并:
- 官方逐专业计数表:`seu_stat`(普通 115 键 3051 人 + 强基 12 键 57 人)、`scut_plan`(统考可用计划 138 行 3322 人)。
- 逐人名单:北师、华东师大、西北工业、南理工、东北大学、北中医、矿大(北京)、上海师大、合肥工业,按(学院, 学院代码, 专业代码, 专业名称, 层次, 学习方式)分组行数。
- 无文字层扫描件:`scut_bs`(8 页)、`scut_ss`(60 页)缓存为空,判为需 OCR,本轮没有强行造数。
- 新表 `y2027_tuimian_2027`(1436 行,11 校)与 `y2027_plan_2027`(138 行);未改动既有 `y2027_tuimian_major`,避免与并行会话相互覆盖。
- 三处会导致假数的判据已修正:
- 汇总/合计行先排除再分组(东南计数表 34 行、强基表 8 行属院系级合计)。
- 专项计划行不计入推免接收人数(东南 33 条研究生支教团、北中医 13 条),否则合计虚高。
- PDF 抽出的 Kangxi 兼容部首(`⼈`/`⼯`/`⽣`)折回正规汉字,全角标点保持原样;这一改动使跨源关联键不再静默漏配,库内残留兼容部首计数为 0。
- 学院列两种错法都处理了:`拟录取学院码及名称` 合并表头拆列后取到的是代码,先保代码再由内容定位补名称;院系列是合并单元格时向下填充。
- 口径冲突留证:东南大学同一批数据既有官方计数表又有逐人名单,名单分组比官方计数多 278 人(差值随专项排除后仍在),判为口径差异(官方「录取数」对名单「拟录取公示」),不判谁错。按规则只入官方计数,182 行推导值丢弃不对账入库。
- 校验器先被故意改坏验证过一轮:非整数、超上界、空专业键、同业务键重复四类各造一条,全部拒绝且探针行未落库;首轮暴露备份脚本取错 `description` 下标(列名在 `d[0]` 不是 `d[1]`),重做后回读核验行数与首行校名一致。
## 产物与证据
- 解析与入库脚本:`运行暂存/agent_ingest_20261031/` 下 `cache_pdf_grid.py`、`parse_round2_from_cache.py`、`finalize_round2.py`、`ingest_round2_roster.py`。
- 候选与诊断:`cand_round2_pdf.json`、`cand_round2_final.json`、`cand_hfut.json`、`round2_pdf_diag.txt`、`seu_crosscheck.txt`、`postcheck_round2.txt`、`spotcheck_round2.txt`。
- 表格缓存 `grid_cache/`(44 份 JSON,2.5M);原始 PDF 未改动。
- 入库前备份 `运行暂存/入库前备份/20261101_tuimian_round2_pre/`:`y2027_tuimian_major.csv.gz`(1587 行,SHA-256 `f971a16b…`)+ 回读核验过的 `receipt.json`。
- `dim_table_meta` 登记 2 表,`dim_field_semantics` 登记 9 条字段语义,`changelog` 2 条。
## 删除边界(提交 Git 后执行)
- 可删:`运行暂存/y2027_tuimian_round2/raw` 中已解析入库的 15 份 PDF 与 `hfut_page_718c76.htm`;此前 A 级已入库并沉淀回执的批次(新东方、中公、启航、高顿、报录比、0856 目录、no_channel、score_lines 等,约 615M)。
- 暂不可删:`运行暂存/official_roster_2025_2026/raw`(106M,29 份 PDF + 47 HTML 尚未解析入库);`运行暂存/scut_bs`、`scut_ss` 两份扫描件是唯一原件且未 OCR。
- 不动:并行会话的 `审计报告/local_ingest_20261031/prewrite_snapshot/` 28G、`_归档/_待删_20261022` 4.9G、`ggyx_chrome/` 登录态。
## 未完成
1. `official_roster_2025_2026` 的 2025/2026 复试与拟录取名单还没解析(表格缓存已就绪),解析前不能删原件。
2. 两份华南理工扫描件需 OCR 才有数值。
3. 本轮新表未接入交付层,`y2027_tuimian_major` 与新表需在交付层按 basis 分列展示。
@@ -0,0 +1,779 @@
[
{
"college": "机械工程学院",
"major_code": "085501",
"major_name": "机械工程",
"kind": "硕士",
"value": 113
},
{
"college": "电气与自动化工程学院",
"major_code": "085801",
"major_name": "电气工程",
"kind": "硕士",
"value": 102
},
{
"college": "管理与智能科学学院",
"major_code": "120100",
"major_name": "管理科学与工程",
"kind": "硕士",
"value": 87
},
{
"college": "电气与自动化工程学院",
"major_code": "080800",
"major_name": "电气工程",
"kind": "硕士",
"value": 55
},
{
"college": "计算机与信息学院",
"major_code": "081200",
"major_name": "计算机科学与技术",
"kind": "硕士",
"value": 49
},
{
"college": "计算机与信息学院",
"major_code": "085410",
"major_name": "人工智能",
"kind": "硕士",
"value": 42
},
{
"college": "数学学院",
"major_code": "070100",
"major_name": "数学",
"kind": "硕士",
"value": 41
},
{
"college": "仪器科学与光电工程学院",
"major_code": "085407",
"major_name": "仪器仪表工程",
"kind": "硕士",
"value": 39
},
{
"college": "食品与生物工程学院",
"major_code": "086003",
"major_name": "食品工程",
"kind": "硕士",
"value": 38
},
{
"college": "计算机与信息学院",
"major_code": "085404",
"major_name": "计算机技术",
"kind": "硕士",
"value": 36
},
{
"college": "微电子学院",
"major_code": "085401",
"major_name": "新一代电子信息技术(含量子技术等)",
"kind": "硕士",
"value": 36
},
{
"college": "管理与智能科学学院",
"major_code": "120200",
"major_name": "工商管理学",
"kind": "硕士",
"value": 35
},
{
"college": "食品与生物工程学院",
"major_code": "083200",
"major_name": "食品科学与工程",
"kind": "硕士",
"value": 35
},
{
"college": "经济学院",
"major_code": "025100",
"major_name": "金融",
"kind": "硕士",
"value": 34
},
{
"college": "材料科学与工程学院",
"major_code": "080500",
"major_name": "材料科学与工程",
"kind": "硕士",
"value": 33
},
{
"college": "材料科学与工程学院",
"major_code": "085601",
"major_name": "材料工程",
"kind": "硕士",
"value": 33
},
{
"college": "仪器科学与光电工程学院",
"major_code": "080400",
"major_name": "仪器科学与技术",
"kind": "硕士",
"value": 30
},
{
"college": "电气与自动化工程学院",
"major_code": "085406",
"major_name": "控制工程",
"kind": "硕士",
"value": 30
},
{
"college": "汽车与交通工程学院",
"major_code": "085502",
"major_name": "车辆工程",
"kind": "硕士",
"value": 27
},
{
"college": "计算机与信息学院",
"major_code": "085402",
"major_name": "通信工程(含宽带网络、移动通信等)",
"kind": "硕士",
"value": 26
},
{
"college": "土木与水利工程学院",
"major_code": "081400",
"major_name": "土木工程",
"kind": "硕士",
"value": 26
},
{
"college": "机械工程学院",
"major_code": "080201",
"major_name": "机械制造及其自动化",
"kind": "硕士",
"value": 22
},
{
"college": "机械工程学院",
"major_code": "080202",
"major_name": "机械电子工程",
"kind": "硕士",
"value": 22
},
{
"college": "机械工程学院",
"major_code": "080203",
"major_name": "机械设计及理论",
"kind": "硕士",
"value": 22
},
{
"college": "微电子学院",
"major_code": "080900",
"major_name": "电子科学与技术",
"kind": "硕士",
"value": 22
},
{
"college": "马克思主义学院",
"major_code": "030500",
"major_name": "马克思主义理论",
"kind": "硕士",
"value": 22
},
{
"college": "化学与化工学院",
"major_code": "081700",
"major_name": "化学工程与技术",
"kind": "硕士",
"value": 21
},
{
"college": "化学与化工学院",
"major_code": "085602",
"major_name": "化学工程",
"kind": "硕士",
"value": 19
},
{
"college": "土木与水利工程学院",
"major_code": "085901",
"major_name": "土木工程",
"kind": "硕士",
"value": 19
},
{
"college": "管理与智能科学学院",
"major_code": "120201",
"major_name": "会计学",
"kind": "硕士",
"value": 19
},
{
"college": "汽车与交通工程学院",
"major_code": "086100",
"major_name": "交通运输",
"kind": "硕士",
"value": 18
},
{
"college": "电气与自动化工程学院",
"major_code": "081100",
"major_name": "控制科学与工程",
"kind": "硕士",
"value": 17
},
{
"college": "计算机与信息学院",
"major_code": "081000",
"major_name": "信息与通信工程",
"kind": "硕士",
"value": 17
},
{
"college": "卓越工程师学院",
"major_code": "085502",
"major_name": "车辆工程",
"kind": "硕士",
"value": 17
},
{
"college": "经济学院",
"major_code": "020200",
"major_name": "应用经济学",
"kind": "硕士",
"value": 14
},
{
"college": "卓越工程师学院",
"major_code": "085501",
"major_name": "机械工程",
"kind": "硕士",
"value": 14
},
{
"college": "外国语学院",
"major_code": "055101",
"major_name": "英语笔译",
"kind": "硕士",
"value": 13
},
{
"college": "汽车与交通工程学院",
"major_code": "080204",
"major_name": "车辆工程",
"kind": "硕士",
"value": 13
},
{
"college": "卓越工程师学院",
"major_code": "085801",
"major_name": "电气工程",
"kind": "硕士",
"value": 13
},
{
"college": "仪器科学与光电工程学院",
"major_code": "083100",
"major_name": "生物医学工程",
"kind": "硕士",
"value": 12
},
{
"college": "化学与化工学院",
"major_code": "070300",
"major_name": "化学",
"kind": "硕士",
"value": 12
},
{
"college": "建筑与艺术学院",
"major_code": "135700",
"major_name": "设计",
"kind": "硕士",
"value": 12
},
{
"college": "食品与生物工程学院",
"major_code": "071000",
"major_name": "生物学",
"kind": "硕士",
"value": 12
},
{
"college": "汽车与交通工程学院",
"major_code": "085802",
"major_name": "动力工程",
"kind": "硕士",
"value": 12
},
{
"college": "集成电路创新学院",
"major_code": "085401",
"major_name": "新一代电子信息技术(含量子技术等)",
"kind": "硕士",
"value": 12
},
{
"college": "智能软件工程学院",
"major_code": "085405",
"major_name": "软件工程",
"kind": "硕士",
"value": 12
},
{
"college": "机械工程学院",
"major_code": "0802Z1",
"major_name": "工业工程",
"kind": "硕士",
"value": 11
},
{
"college": "土木与水利工程学院",
"major_code": "080100",
"major_name": "力学",
"kind": "硕士",
"value": 11
},
{
"college": "外国语学院",
"major_code": "050200",
"major_name": "外国语言文学",
"kind": "硕士",
"value": 11
},
{
"college": "汽车与交通工程学院",
"major_code": "082300",
"major_name": "交通运输工程",
"kind": "硕士",
"value": 10
},
{
"college": "经济学院",
"major_code": "020100",
"major_name": "理论经济学",
"kind": "硕士",
"value": 10
},
{
"college": "文法学院",
"major_code": "035102",
"major_name": "法律(法学)",
"kind": "硕士",
"value": 10
},
{
"college": "卓越工程师学院",
"major_code": "085601",
"major_name": "材料工程",
"kind": "硕士",
"value": 10
},
{
"college": "卓越工程师学院",
"major_code": "085500",
"major_name": "机械",
"kind": "直博生",
"value": 10
},
{
"college": "土木与水利工程学院",
"major_code": "085902",
"major_name": "水利工程",
"kind": "硕士",
"value": 8
},
{
"college": "土木与水利工程学院",
"major_code": "085905",
"major_name": "市政工程(含给排水等)",
"kind": "硕士",
"value": 8
},
{
"college": "资源与环境工程学院",
"major_code": "085701",
"major_name": "环境工程",
"kind": "硕士",
"value": 8
},
{
"college": "人工智能创新学院",
"major_code": "085410",
"major_name": "人工智能",
"kind": "硕士",
"value": 8
},
{
"college": "土木与水利工程学院",
"major_code": "081500",
"major_name": "水利工程",
"kind": "硕士",
"value": 7
},
{
"college": "资源与环境工程学院",
"major_code": "070500",
"major_name": "地理学",
"kind": "硕士",
"value": 7
},
{
"college": "资源与环境工程学院",
"major_code": "083000",
"major_name": "环境科学与工程",
"kind": "硕士",
"value": 7
},
{
"college": "食品与生物工程学院",
"major_code": "105500",
"major_name": "药学",
"kind": "硕士",
"value": 7
},
{
"college": "汽车与交通工程学院",
"major_code": "080700",
"major_name": "动力工程及工程热物理",
"kind": "硕士",
"value": 7
},
{
"college": "聚变科学与工程学院",
"major_code": "085402",
"major_name": "通信工程(含宽带网络、移动通信等)",
"kind": "硕士",
"value": 7
},
{
"college": "聚变科学与工程学院",
"major_code": "085801",
"major_name": "电气工程",
"kind": "硕士",
"value": 7
},
{
"college": "土木与水利工程学院",
"major_code": "081403",
"major_name": "市政工程",
"kind": "硕士",
"value": 6
},
{
"college": "建筑与艺术学院",
"major_code": "086200",
"major_name": "风景园林",
"kind": "硕士",
"value": 6
},
{
"college": "食品与生物工程学院",
"major_code": "0817Z1",
"major_name": "制药工程",
"kind": "硕士",
"value": 6
},
{
"college": "卓越工程师学院",
"major_code": "085410",
"major_name": "人工智能",
"kind": "硕士",
"value": 6
},
{
"college": "集成电路创新学院",
"major_code": "0812J1",
"major_name": "集成电路与系统",
"kind": "硕士",
"value": 6
},
{
"college": "土木与水利工程学院",
"major_code": "081404",
"major_name": "供热、供燃气、通风及空调工程",
"kind": "硕士",
"value": 5
},
{
"college": "建筑与艺术学院",
"major_code": "083300",
"major_name": "城乡规划学",
"kind": "硕士",
"value": 5
},
{
"college": "建筑与艺术学院",
"major_code": "085100",
"major_name": "建筑",
"kind": "硕士",
"value": 5
},
{
"college": "建筑与艺术学院",
"major_code": "085507",
"major_name": "工业设计工程",
"kind": "硕士",
"value": 5
},
{
"college": "建筑与艺术学院",
"major_code": "130100",
"major_name": "艺术学",
"kind": "硕士",
"value": 5
},
{
"college": "建筑与艺术学院",
"major_code": "140300",
"major_name": "设计学",
"kind": "硕士",
"value": 5
},
{
"college": "经济学院",
"major_code": "025400",
"major_name": "国际商务",
"kind": "硕士",
"value": 5
},
{
"college": "物理学院",
"major_code": "085401",
"major_name": "新一代电子信息技术(含量子技术等)",
"kind": "硕士",
"value": 5
},
{
"college": "卓越工程师学院",
"major_code": "085406",
"major_name": "控制工程",
"kind": "硕士",
"value": 5
},
{
"college": "人工智能创新学院",
"major_code": "081200",
"major_name": "计算机科学与技术",
"kind": "硕士",
"value": 5
},
{
"college": "聚变科学与工程学院",
"major_code": "085501",
"major_name": "机械工程",
"kind": "硕士",
"value": 5
},
{
"college": "智能软件工程学院",
"major_code": "081200",
"major_name": "计算机科学与技术",
"kind": "硕士",
"value": 5
},
{
"college": "资源与环境工程学院",
"major_code": "081800",
"major_name": "地质资源与地质工程",
"kind": "硕士",
"value": 4
},
{
"college": "食品与生物工程学院",
"major_code": "086001",
"major_name": "生物技术与工程",
"kind": "硕士",
"value": 4
},
{
"college": "食品与生物工程学院",
"major_code": "086002",
"major_name": "制药工程",
"kind": "硕士",
"value": 4
},
{
"college": "物理学院",
"major_code": "070200",
"major_name": "物理学",
"kind": "硕士",
"value": 4
},
{
"college": "文法学院",
"major_code": "030100",
"major_name": "法学",
"kind": "硕士",
"value": 4
},
{
"college": "聚变科学与工程学院",
"major_code": "070200",
"major_name": "物理学",
"kind": "硕士",
"value": 4
},
{
"college": "卓越工程师学院",
"major_code": "085800",
"major_name": "能源动力",
"kind": "直博生",
"value": 4
},
{
"college": "化学与化工学院",
"major_code": "080502",
"major_name": "材料学",
"kind": "硕士",
"value": 3
},
{
"college": "化学与化工学院",
"major_code": "085601",
"major_name": "材料工程",
"kind": "硕士",
"value": 3
},
{
"college": "卓越工程师学院",
"major_code": "085401",
"major_name": "新一代电子信息技术(含量子技术等)",
"kind": "硕士",
"value": 3
},
{
"college": "卓越工程师学院",
"major_code": "085404",
"major_name": "计算机技术",
"kind": "硕士",
"value": 3
},
{
"college": "聚变科学与工程学院",
"major_code": "081000",
"major_name": "信息与通信工程",
"kind": "硕士",
"value": 3
},
{
"college": "卓越工程师学院",
"major_code": "085400",
"major_name": "电子信息",
"kind": "直博生",
"value": 3
},
{
"college": "机械工程学院",
"major_code": "085802",
"major_name": "动力工程",
"kind": "硕士",
"value": 2
},
{
"college": "土木与水利工程学院",
"major_code": "081600",
"major_name": "测绘科学与技术",
"kind": "硕士",
"value": 2
},
{
"college": "土木与水利工程学院",
"major_code": "085704",
"major_name": "测绘工程",
"kind": "硕士",
"value": 2
},
{
"college": "土木与水利工程学院",
"major_code": "085906",
"major_name": "人工环境工程(含供热、通风及空调等)",
"kind": "硕士",
"value": 2
},
{
"college": "建筑与艺术学院",
"major_code": "081300",
"major_name": "建筑学",
"kind": "硕士",
"value": 2
},
{
"college": "聚变科学与工程学院",
"major_code": "085601",
"major_name": "材料工程",
"kind": "硕士",
"value": 2
},
{
"college": "电气与自动化工程学院",
"major_code": "085800",
"major_name": "能源动力",
"kind": "直博生",
"value": 2
},
{
"college": "管理与智能科学学院",
"major_code": "085400",
"major_name": "电子信息",
"kind": "直博生",
"value": 2
},
{
"college": "资源与环境工程学院",
"major_code": "085703",
"major_name": "地质工程",
"kind": "硕士",
"value": 1
},
{
"college": "物理学院",
"major_code": "080300",
"major_name": "光学工程",
"kind": "硕士",
"value": 1
},
{
"college": "卓越工程师学院",
"major_code": "085407",
"major_name": "仪器仪表工程",
"kind": "硕士",
"value": 1
},
{
"college": "材料科学与工程学院",
"major_code": "080500",
"major_name": "材料科学与工程",
"kind": "直博生",
"value": 1
},
{
"college": "电气与自动化工程学院",
"major_code": "085400",
"major_name": "电子信息",
"kind": "直博生",
"value": 1
},
{
"college": "资源与环境工程学院",
"major_code": "070900",
"major_name": "地质学",
"kind": "直博生",
"value": 1
},
{
"college": "管理与智能科学学院",
"major_code": "085500",
"major_name": "机械",
"kind": "直博生",
"value": 1
},
{
"college": "人工智能创新学院",
"major_code": "081200",
"major_name": "计算机科学与技术",
"kind": "直博生",
"value": 1
}
]
@@ -0,0 +1,16 @@
Consider using the pymupdf_layout package for a greatly improved page layout analysis.
tm27_bnu_pdf_cbaa1326 pages=70 bytes=250812
tm27_bucm_pdf2_0412bf pages=16 bytes=68023
tm27_cumtb_pdf_0c26f3 pages=15 bytes=82605
tm27_ecnu_pdf_ea6612af pages=50 bytes=232930
tm27_neu_pdf_2293fa87 pages=116 bytes=361225
tm27_njust_pdf_3fed60bb pages=80 bytes=175114
tm27_nwpu_pdf_466e2335 pages=46 bytes=278807
tm27_scut_bs_5698d74a pages=8 bytes=75
tm27_scut_plan_38aada94 pages=3 bytes=8121
tm27_scut_ss_f2e9d15f pages=60 bytes=283
tm27_seu_qj_79a99de5 pages=3 bytes=10833
tm27_seu_ss_c31b26d0 pages=89 bytes=446893
tm27_seu_stat_8bab9f8e pages=3 bytes=10361
tm27_seu_stat_qj_8a75a02e pages=1 bytes=1369
tm27_shnu_pdf_2fce5d pages=5 bytes=18164
@@ -0,0 +1,35 @@
{
"generated_at": "2026-11-01",
"rows_in_cand": 1574,
"tables": {
"y2027_tuimian_2027": 1436,
"y2027_plan_2027": 138
},
"source_files_ingested": [
"15份官方推免公示PDF",
"hfut_page_718c76.htm"
],
"source_files_not_ingested": {
"scanned_no_text_layer": [
"scut_bs_5698d74a.pdf",
"scut_ss_f2e9d15f.pdf"
],
"entry_pages_without_table": "其余63个HTML多数是入口页,名单在PDF附件",
"dropped_after_crosscheck": {
"seu_ss_c31b26d0.pdf/seu_qj_79a99de5.pdf": "同校已有官方计数表,182行推导值只作对账"
}
},
"backup_sha256": "f971a16b1a577e07e1a2e9a56e44706ed22dbf50a40997d2264868376c167a8c",
"files": {
"cand_round2_final.json.gz": "73b6f9d678f6091c83d0267c9ae6a42787aa0589c95598d05d322ffade3fc9b5",
"hfut_2027_tuimian.json": "748bee8f0e1818d9e47139ce35c855fd22bb2aa3ec260c0f35f4cdd391d747fb",
"log_cache_tm27.txt": "1665d9af4bcbf32d2c714c9dab6778cf1e2021c9c1917208c9d184ae31ef95d6",
"postcheck_round2.txt": "55c91e37e8e06a9e88cacf32dd190db70e8eb3055e5259e99b5bc76ee07ab0cf",
"receipt.json": "34b9a0412e021d04b72d5cf2fbd2062eabd393302e6c158e0421529bb5f0f840",
"round2_pdf_diag.txt": "12673e14b0c13b6923dcf2879701345d6595af2da158e1a5fb6e152d7deee438",
"round2_tag_index.json": "069bc2beb319c596e86913850f31235033138be93a1fa30dac5b010cf5f0222a",
"seu_crosscheck.txt": "057a29538585e580c659b3da9888e2834ff449e5b0262cf7b4ce1ea482cb86b9",
"spotcheck_round2.txt": "47644165d81b5e4c483e51d67fc1ba0462f66f6c98f3b279c5826f9980c3cc11",
"y2027_tuimian_major.csv.gz": "f971a16b1a577e07e1a2e9a56e44706ed22dbf50a40997d2264868376c167a8c"
}
}
@@ -0,0 +1,31 @@
=== 入库后核验 y2027_tuimian_2027 ===
上海师范大学 roster_derived 键=55 人数=271 区间=1~24
东北大学 roster_derived 键=99 人数=2307 区间=1~164
东南大学 count_direct 键=127 人数=3108 区间=1~190
中国矿业大学(北京) roster_derived 键=71 人数=624 区间=1~80
北京中医药大学 roster_derived 键=68 人数=405 区间=1~81
北京师范大学 roster_derived 键=225 人数=2268 区间=1~105
华东师范大学 roster_derived 键=365 人数=2429 区间=1~107
南京理工大学 roster_derived 键=128 人数=1832 区间=1~140
合肥工业大学 roster_derived 键=111 人数=1679 区间=1~113
西北工业大学 roster_derived 键=187 人数=2702 区间=1~124
总键=1436 总人数=17625
值域异常(<=0 或 >2000): 0
非数字值: 0
空专业键(code与name都空): 0
自设码(含字母): 24
残留CJK兼容部首: 0
plan_type 分布: [('强基计划', 12), ('普通推免', 1424)]
=== y2027_plan_2027 ===
('华南理工大学', 138, 3322)
空名称: 0
=== 候选 vs 库内 键数对账 ===
上海师范大学 候选=55 库=55 OK
东北大学 候选=99 库=99 OK
东南大学 候选=127 库=127 OK
中国矿业大学(北京) 候选=71 库=71 OK
北京中医药大学 候选=68 库=68 OK
北京师范大学 候选=225 库=225 OK
华东师范大学 候选=365 库=365 OK
南京理工大学 候选=128 库=128 OK
西北工业大学 候选=187 库=187 OK
@@ -0,0 +1,8 @@
{
"table": "y2027_tuimian_major",
"rows_before": 1587,
"sha256": "f971a16b1a577e07e1a2e9a56e44706ed22dbf50a40997d2264868376c167a8c",
"readback_rows": 1587,
"readback_first_school": "东华大学",
"at": "2026-11-01"
}
@@ -0,0 +1,15 @@
bnu 北京师范大学 hdr=1832 total_skip=0 nohm=1 行=49 直接计数=0 分组键=49
bucm 北京中医药大学 hdr=204 total_skip=0 nohm=0 行=1 直接计数=0 分组键=1
cumtb 中国矿业大学(北京) hdr=607 total_skip=0 nohm=1 行=0 直接计数=0 分组键=0
ecnu 华东师范大学 hdr=1741 total_skip=0 nohm=1 行=83 直接计数=0 分组键=83
neu 东北大学 hdr=2361 total_skip=0 nohm=0 行=5 直接计数=0 分组键=5
njust 南京理工大学 hdr=1821 total_skip=0 nohm=0 行=2 直接计数=0 分组键=2
nwpu 西北工业大学 hdr=2538 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0
scut 华南理工大学 hdr=0 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0
scut 华南理工大学 hdr=38 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0
scut 华南理工大学 hdr=0 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0
seu 东南大学 hdr=80 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0
seu 东南大学 hdr=3222 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0
seu 东南大学 hdr=101 total_skip=3 nohm=0 行=0 直接计数=0 分组键=0
seu 东南大学 hdr=13 total_skip=0 nohm=0 行=0 直接计数=0 分组键=0
shnu 上海师范大学 hdr=5 total_skip=0 nohm=0 行=55 直接计数=0 分组键=55
@@ -0,0 +1,322 @@
{
"scu_list": {
"school": "四川大学",
"url": "https://yz.scu.edu.cn/zsxx/newslist/a/gk"
},
"bucm_page": {
"school": "北京中医药大学",
"url": "https://yanjiusheng.bucm.edu.cn/zsjy/sszs/28766041e1154bd998292bab68a0ce32.htm"
},
"uir_home": {
"school": "国际关系学院",
"url": "https://yjszs.uir.cn/"
},
"tju_201": {
"school": "天津大学-机械",
"url": "https://me.tju.edu.cn/info/1041/54111.htm"
},
"tju_202": {
"school": "天津大学-精仪",
"url": "https://jyxy.tju.edu.cn/cn/new/20260928/2499.shtml"
},
"tju_209": {
"school": "天津大学-经管",
"url": "https://come.tju.edu.cn/info/1013/18623.htm"
},
"cumtb_pdf": {
"school": "中国矿业大学(北京)",
"url": "https://yz.cumtb.edu.cn/upload/uid1/b/0f/c3/98ac7032308e92d40608b2bd4dfa178d5cec30fb.pdf"
},
"ncu_list": {
"school": "南昌大学",
"url": "http://yjsy.ncu.edu.cn/index/gszl.htm"
},
"tyut_page": {
"school": "太原理工大学",
"url": "http://www.gs.tyut.edu.cn/info/1261/16789.htm"
},
"ncu_page2": {
"school": "南昌大学",
"url": "http://yjsy.ncu.edu.cn/info/1014/47886.htm"
},
"shnu_page": {
"school": "上海师范大学",
"url": "https://yjsc.shnu.edu.cn/1e/15/c17206a859669/page.htm"
},
"hunnu_list": {
"school": "湖南师范大学",
"url": "https://yjsy.hunnu.edu.cn/zsks/sszk1/zkdt1.htm"
},
"shnu_pdf": {
"school": "上海师范大学",
"url": "https://yjsc.shnu.edu.cn/_upload/article/files/7c/c0/0259cecb4b26b37cc4f095d20dbf/6251b6ea-1dec-4b7d-a400-9e4a00678a82.pdf"
},
"hunnu_page2": {
"school": "湖南师范大学",
"url": "https://yjsy.hunnu.edu.cn/info/1028/16375.htm"
},
"scu_sample": {
"school": "四川大学",
"url": "https://yz.scu.edu.cn/tms/tmsgs?yxs=101"
},
"chsi_nju": {
"school": "南京大学",
"url": "https://yz.chsi.com.cn/sswbgg/pages/msgdetail.do?dwdm=10284&msg_id=574095"
},
"cumt_list": {
"school": "中国矿业大学",
"url": "https://yz.cumt.edu.cn/zsxx/tzgg.htm"
},
"hunnu_js": {
"school": "湖南师范大学",
"url": "https://yjsy.hunnu.edu.cn/zsks/sszk1/zkdt1/16375.htm"
},
"bnu_page": {
"school": "北京师范大学",
"url": "https://yz.bnu.edu.cn/detail/2bdb9414-87e7-4509-b151-caba9e2ae4c5"
},
"ecust_page": {
"school": "华东理工大学",
"url": "https://gschool.ecust.edu.cn/2026/1021/c12750a194415/page.htm"
},
"scu_yxstmgs": {
"school": "四川大学",
"url": "https://yz.scu.edu.cn/tms/yxstmgs"
},
"scu_jkgs": {
"school": "四川大学",
"url": "https://yz.scu.edu.cn/tms/tmsjkgs"
},
"nju_home": {
"school": "南京大学",
"url": "https://yzb.nju.edu.cn/"
},
"nju_news": {
"school": "南京大学",
"url": "https://yzb.nju.edu.cn/4794/list.htm"
},
"nju_stat": {
"school": "南京大学",
"url": "https://yzb.nju.edu.cn/ea/44/c47863a846404/page.htm"
},
"scu_yxs2": {
"school": "四川大学",
"url": "https://yz.scu.edu.cn/tms/yxstmgs"
},
"neu_page": {
"school": "东北大学",
"url": "https://yz.neu.edu.cn/2026/1020/c5792a480951/page.htm"
},
"dlut_page": {
"school": "大连理工",
"url": "https://gs.dlut.edu.cn/info/1173/52122.htm"
},
"jlu_home": {
"school": "吉林大学",
"url": "https://yz.jlu.edu.cn/"
},
"neu_page2": {
"school": "东北大学",
"url": "https://yz.neu.edu.cn/2026/1020/c5792a480951/page.htm"
},
"dlut_page2": {
"school": "大连理工",
"url": "https://gs.dlut.edu.cn/info/1173/52122.htm"
},
"nwpu_page": {
"school": "西北工业大学",
"url": "https://yzb.nwpu.edu.cn/info/1174/10568.htm"
},
"njust_page": {
"school": "南京理工大学",
"url": "https://gs.njust.edu.cn/zsw/b3/aa/c4587a373674/page.htm"
},
"bucm_pdf2": {
"school": "北京中医药大学",
"url": "https://yanjiusheng.bucm.edu.cn/docs/2026-10/33ea8b9a9ff045b08ec501ae8e5e74d1.pdf"
},
"bjtu_page": {
"school": "北京交通大学",
"url": "https://yzb.bjtu.edu.cn/tzgg/134792e073ba4910b2f225d6068abd5d.htm"
},
"ncepu_bj": {
"school": "华北电力大学",
"url": "https://yjsy.ncepu.edu.cn/zsxx/sszsxx/556a6dea89a8456aae0dbfada9811d11.htm"
},
"ncepu_bd": {
"school": "华北电力大学(保定)",
"url": "https://gs.ncepu.edu.cn/zsxx/cxfw/7a74a16edf264830a215194d18436229.htm"
},
"zzu_home": {
"school": "郑州大学",
"url": "https://yz.zzu.edu.cn/"
},
"zzu_page2": {
"school": "郑州大学",
"url": "https://yz.zzu.edu.cn/info/1070/2436.htm"
},
"cumt_page": {
"school": "中国矿业大学",
"url": "https://yz.cumt.edu.cn/info/1004/2615.htm"
},
"cumtb_list": {
"school": "中国矿业大学(北京)",
"url": "https://yz.cumtb.edu.cn/index/newslist.html?type=1&class_id=156"
},
"bjtu_home": {
"school": "北京交通大学",
"url": "https://yzb.bjtu.edu.cn/"
},
"zzu_page3": {
"school": "郑州大学",
"url": "https://yz.zzu.edu.cn/info/1070/2436.htm"
},
"cumt_page2": {
"school": "中国矿业大学",
"url": "https://yz.cumt.edu.cn/info/1004/2615.htm"
},
"cumtb_list2": {
"school": "中国矿业大学(北京)",
"url": "https://yz.cumtb.edu.cn/index/newslist.html?type=1&class_id=156"
},
"hfut_page": {
"school": "合肥工业大学",
"url": "https://yjszs.hfut.edu.cn/2026/1015/c13533a323798/pagem.htm"
},
"fzu_list": {
"school": "福州大学",
"url": "https://yjsy.fzu.edu.cn/index/gsl/gsl.htm"
},
"zzu_pdf": {
"school": "郑州大学",
"url": "https://yz.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1876580606&wbfileid=18210509"
},
"cumtb_page": {
"school": "中国矿业大学(北京)",
"url": "https://yz.cumtb.edu.cn/index/detail.html?id=06ad8b9a960651"
},
"fzu_page": {
"school": "福州大学",
"url": "https://yjsy.fzu.edu.cn/info/1077/6211.htm"
},
"bnu_pdf": {
"school": "北京师范大学",
"url": "https://yz.bnu.edu.cn/content/docs/ss_nlqmd/2027%E5%85%AC%E7%A4%BA%E6%8E%A8%E5%85%8D%E6%8B%9F%E5%BD%95%E5%8F%96%E5%90%8D%E5%8D%95.pdf"
},
"ecnu_pdf": {
"school": "华东师大",
"url": "https://yjszs.ecnu.edu.cn/_upload/article/files/fd/42/0b3b95dd435091470426d3262526/02854f30-1e4a-4ad3-b38a-80b5afa5c1b9.pdf"
},
"ncu_page": {
"school": "南昌大学",
"url": "http://yjsy.ncu.edu.cn/info/1014/47886.htm"
},
"neu_pdf": {
"school": "东北大学",
"url": "https://yz.neu.edu.cn/_upload/article/files/31/d8/12345b4c4433b7c87f1761a8e1d3/e23c36d2-64ad-4d3c-a765-b57c30276adc.pdf"
},
"dlut_pdf": {
"school": "大连理工",
"url": "https://gs.dlut.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1502106601&wbfileid=1C922D0828D21F37D177FBD370C8E2C0"
},
"nwpu_pdf": {
"school": "西北工业大学",
"url": "https://yzb.nwpu.edu.cn/__local/6/88/AA/EA4062A7A3EE1A67139DC748D75_4F482E24_ABA6D.pdf"
},
"njust_pdf": {
"school": "南京理工大学",
"url": "https://gs.njust.edu.cn/_upload/article/files/d3/20/f77f1f2b46a18a938b187549c9bc/be959d85-dd4f-4794-9ec4-336ff2b32000.pdf"
},
"ecnu": {
"school": "华东师大",
"url": "https://yjszs.ecnu.edu.cn/tzgg_43463/list.htm"
},
"csu": {
"school": "中南",
"url": "https://yz.csu.edu.cn/"
},
"xmu": {
"school": "厦门大学",
"url": "https://zs.xmu.edu.cn/sss.htm"
},
"hnu": {
"school": "湖南大学",
"url": "https://gra.hnu.edu.cn/index/zsxx.htm"
},
"bit": {
"school": "北京理工",
"url": "https://yzb.bit.edu.cn/"
},
"nuaa": {
"school": "南航",
"url": "https://yzb.nuaa.edu.cn/"
},
"tongji": {
"school": "同济",
"url": "https://yz.tongji.edu.cn/zsxw/ggtz.htm"
},
"uestc": {
"school": "电子科大",
"url": "https://yz.uestc.edu.cn/sszs/tms.htm"
},
"hust": {
"school": "华科",
"url": "https://gszs.hust.edu.cn/zsgg/sszs.htm"
},
"fudan2": {
"school": "复旦",
"url": "https://gsao.fudan.edu.cn/15020/list2.htm"
},
"ecnu_page": {
"school": "华东师大",
"url": "https://yjszs.ecnu.edu.cn/0c/6d/c43463a789613/page.htm"
},
"nuaa_page": {
"school": "南京航空航天大学",
"url": "https://www.graduate.nuaa.edu.cn/2026/1012/c13502a414577/page.htm"
},
"xmu_tms": {
"school": "厦门大学",
"url": "https://zs.xmu.edu.cn/sss/zsjz/tms.htm"
},
"csu_tm": {
"school": "中南",
"url": "https://yz.csu.edu.cn/tms.htm"
},
"hnu_list": {
"school": "湖南大学",
"url": "https://gra.hnu.edu.cn/index/zsxx.htm"
},
"scut_ss": {
"school": "华南理工大学",
"url": "https://yz.scut.edu.cn/_upload/article/files/3e/8a/b77f81504aed9a0e4c99014a6f71/9a0b2657-7f75-4b61-a3d2-e010c414bbf4.pdf"
},
"scut_bs": {
"school": "华南理工大学",
"url": "https://yz.scut.edu.cn/_upload/article/files/3e/8a/b77f81504aed9a0e4c99014a6f71/523dc474-d4f7-4966-8faf-45c6be6feca4.pdf"
},
"scut_plan": {
"school": "华南理工大学",
"url": "https://yz.scut.edu.cn/_upload/article/files/3e/8a/b77f81504aed9a0e4c99014a6f71/e89c2239-c5ac-463d-b3fb-94b95fa53492.pdf"
},
"seu_ss": {
"school": "东南大学",
"url": "https://yzb.seu.edu.cn/_upload/article/files/31/38/fa8cc19649cfa91ba1dd178b4807/0377af62-fc21-4647-bdb8-0aeb862ddd00.pdf"
},
"seu_qj": {
"school": "东南大学",
"url": "https://yzb.seu.edu.cn/_upload/article/files/31/38/fa8cc19649cfa91ba1dd178b4807/352760b3-26f4-41bc-a4f4-ae04a8f1c082.pdf"
},
"seu_stat": {
"school": "东南大学",
"url": "https://yzb.seu.edu.cn/_upload/article/files/31/38/fa8cc19649cfa91ba1dd178b4807/e2fa7c8b-328f-472d-bca9-1b55a39a41f8.pdf"
},
"seu_stat_qj": {
"school": "东南大学",
"url": "https://yzb.seu.edu.cn/_upload/article/files/31/38/fa8cc19649cfa91ba1dd178b4807/1fe20865-27b2-4b86-b27b-596a5d7dceff.pdf"
},
"zzu_page": {
"school": "郑州大学",
"url": "https://yz.zzu.edu.cn/info/1070/2436.htm"
}
}
@@ -0,0 +1,8 @@
排除专项后:官方计数=3051人 名单分组=3329人
对齐键=115 完全相等=63 占比=54.8%
剩余差异键=52 差值合计=270
(('004', '081000'), 124, 179)
(('404', '085403'), 132, 165)
(('010', '070200'), 22, 39)
(('004', '080904'), 82, 98)
(('004', '085400'), 142, 157)
@@ -0,0 +1,14 @@
--- 抽查 5 行 ---
('东南大学', '能源与环境学院', '089901', '碳中和科学技术', '', '4', 'count_direct')
('北京师范大学', '物理与天文学院', '', '天文学', '博士', '1', 'roster_derived')
('北京师范大学', '物理与天文学院', '', '物理教育', '硕士', '3', 'roster_derived')
('中国矿业大学(北京)', '能源与矿业学院', '0819J4', '储能科学与工程', '硕士', '2', 'roster_derived')
('北京师范大学', '未来教育学院', '', '学科教学(数学)', '硕士', '18', 'roster_derived')
--- 原文比对:西北工业大学 080100 力学 ---
缓存内 080100 名单行数=81 库内值=('5',)
--- 原文比对:合肥工业 085501 ---
候选=[{'college': '机械工程学院', 'major_code': '085501', 'major_name': '机械工程', 'kind': '硕士', 'value': 113}, {'college': '卓越工程师学院', 'major_code': '085501', 'major_name': '机械工程', 'kind': '硕士', 'value': 14}, {'college': '聚变科学与工程学院', 'major_code': '085501', 'major_name': '机械工程', 'kind': '硕士', 'value': 5}] 库内=[('机械与电气工程学院', '42'), ('机械工程与自动化学院', '136'), ('机械工程与自动化学院', '2'), ('机械工程学院', '92'), ('中法工程师学院', '12'), ('机电学院', '27'), ('机械工程学院', '113'), ('卓越工程师学院', '14'), ('聚变科学与工程学院', '5')]
--- 东南大学强基与常规分列 ---
('强基计划', 12, 57)
('普通推免', 115, 3051)
+1 -1
View File
@@ -140,7 +140,7 @@ for idx, t in enumerate(tabs, 1):
except Exception:
pass
profile.append([t, c, typ or '', n, nn, miss, nd, mn, mx])
if miss > 50 and nn > 0 and t not in SEM_WILD:
if miss > 50 and nn > 0 and t not in SEM_WILD and (t, c) not in SEM:
add('P4', '高缺失字段', c, '空值率 %.1f%%' % miss, n - nn)
if is_num and nn:
FLAGGED = "IFNULL(data_flag,'')<>''" if 'data_flag' in cn else '0'
+55
View File
@@ -0,0 +1,55 @@
# -*- coding: utf-8 -*-
"""一次性把名单 PDF 的表格 grid 缓存成 JSON,后续解析迭代只读缓存。
find_tables 在百页级名单上极慢,重复试错等于每次重跑十几分钟,故先缓存。
缓存只含表格单元格的文本,不含图片,体积远小于原件。
"""
import glob
import json
import os
import sys
import fitz
OUTDIR = r'D:/择校/运行暂存/agent_ingest_20261031/grid_cache'
SRC = sys.argv[1] if len(sys.argv) > 1 else r'D:/择校/运行暂存/y2027_tuimian_round2/raw'
TAG = sys.argv[2] if len(sys.argv) > 2 else 'tm27' # 输出文件名前缀,两批目录可能同名
def main():
os.makedirs(OUTDIR, exist_ok=True)
for p in sorted(glob.glob(SRC + '/*.pdf')):
stem = TAG + '_' + os.path.basename(p).rsplit('.', 1)[0]
dst = os.path.join(OUTDIR, stem + '.json')
if os.path.exists(dst):
print('skip', stem)
continue
try:
doc = fitz.open(p)
except Exception as e:
print('FAIL', stem, e)
continue
pages = []
for pi, page in enumerate(doc):
try:
tabs = list(page.find_tables().tables)
except Exception:
tabs = []
grid = []
for tb in tabs:
try:
for row in tb.extract():
grid.append([('' if c is None else str(c)) for c in row])
except Exception:
continue
if not grid: # 无结构表格时退到纯文本行
grid = [[l] for l in page.get_text().split('\n') if l.strip()]
pages.append(grid)
doc.close()
with open(dst, 'w', encoding='utf-8') as f:
json.dump({'file': os.path.basename(p), 'pages': pages}, f, ensure_ascii=False)
print('%-26s pages=%d bytes=%d' % (stem, len(pages), os.path.getsize(dst)))
if __name__ == '__main__':
main()
+59
View File
@@ -0,0 +1,59 @@
# -*- coding: utf-8 -*-
"""合并本轮名单解析候选,按口径规则过滤后入库。
三条过滤规则各有出处:
1. 同一校已有官方逐专业计数表时,名单分组推导值只作对账证据,不入库(东南)。
2. 计划表里落不进计数列的残行不是推免人数,丢弃而不是标 flag(华南理工 2 行)。
3. 专项计划行(支教团等)已在解析阶段排除,此处不重复计数。
"""
import json
DROP_DERIVED = {'seu_ss_c31b26d0.pdf', 'seu_qj_79a99de5.pdf'}
DROP_PLAN_ROWS = {'scut_plan_38aada94.pdf'}
def main():
pdf = json.load(open(r'D:/择校/运行暂存/agent_ingest_20261031/cand_round2_pdf.json',
encoding='utf-8'))
hfut = json.load(open(r'D:/择校/运行暂存/agent_ingest_20261031/cand_hfut.json',
encoding='utf-8'))
for x in hfut:
x.setdefault('college_code', '')
rows = pdf + hfut
kept, dropped = [], {'seu_derived': 0, 'plan_badrow': 0}
for x in rows:
if x['local_file'] in DROP_DERIVED and x.get('basis') == 'roster_derived':
dropped['seu_derived'] += 1
continue
if x['local_file'] in DROP_PLAN_ROWS and x.get('basis') == 'roster_derived':
dropped['plan_badrow'] += 1
continue
note = []
if x['basis'] == 'count_direct':
note.append('官方逐专业计数表' if x.get('target') != 'plan' else '官方统考可用计划表')
else:
note.append('名单分组推导,已排除专项计划行')
if not x.get('major_code'):
note.append('原表无专业代码')
if not x.get('major_name'):
note.append('原表无专业名称')
if x['local_file'].startswith('seu_qj'):
note.append('强基计划单独表')
if x['local_file'].startswith('seu_qj') or x['local_file'].startswith('seu_stat_qj'):
x['plan_type'] = '强基计划'
elif x.get('target') == 'plan':
x['plan_type'] = '统考可用计划'
else:
x['plan_type'] = '普通推免'
x['note'] = ';'.join(note)
x['value'] = str(x['value'])
x.pop('_count_label', None)
kept.append(x)
dst = r'D:/择校/运行暂存/agent_ingest_20261031/cand_round2_final.json'
with open(dst, 'w', encoding='utf-8') as f:
json.dump(kept, f, ensure_ascii=False, indent=1)
print('保留 %d 丢弃 %s' % (len(kept), dropped))
if __name__ == '__main__':
main()
+123
View File
@@ -0,0 +1,123 @@
# -*- coding: utf-8 -*-
"""2027 推免/计划离线解析结果入库(新表,不动既有 y2027_tuimian_major)。
分列口径:
y2027_tuimian_2027.basis = count_direct 原表自带「录取数」列,官方实测
y2027_tuimian_2027.basis = roster_derived 逐人名单分组行数,推导值
y2027_plan_2027 「统考可用计划」,与推免人数不同口径,单表存
入库前核验:代码格式、值为正整数且在上界内、空键拒绝、业务键去重。
入库后核验:行数、按校分布、值域、与源文件行数对账。
"""
import json
import os
import re
import sqlite3
import sys
import time
ZX = r'D:/择校/数据库/zexiao_2027.db'
CODE_OK = re.compile(r'^\d{6}$|^\d{4}[A-Z]\d{1,2}$|^\d{2}[A-Z]\d{3,4}$') # 含字母自设码合法
MAX_COUNT = 3000
DDL_TM = '''CREATE TABLE IF NOT EXISTS y2027_tuimian_2027 (
"school" TEXT, "year" TEXT, "college" TEXT, "college_code" TEXT,
"major_code" TEXT, "major_name" TEXT, "kind" TEXT,
"value" TEXT, "basis" TEXT, "plan_type" TEXT, "source_url" TEXT, "local_file" TEXT,
"note" TEXT, "data_flag" TEXT, "_ingested_at" TEXT,
PRIMARY KEY ("school","year","college","college_code","major_code","major_name",
"kind","basis","plan_type")
)'''
DDL_PLAN = '''CREATE TABLE IF NOT EXISTS y2027_plan_2027 (
"school" TEXT, "year" TEXT, "college" TEXT, "college_code" TEXT,
"major_code" TEXT, "major_name" TEXT, "study_mode" TEXT,
"value" TEXT, "kind" TEXT, "source_url" TEXT, "local_file" TEXT,
"note" TEXT, "data_flag" TEXT, "_ingested_at" TEXT,
PRIMARY KEY ("school","year","college","college_code","major_code","major_name","study_mode","kind")
)'''
def validate(rows, keys):
"""入库前核验,返回 (通过行, 拒绝原因计数)。"""
bad = {}
seen, out = set(), []
for r in rows:
v = str(r.get('value', '')).strip()
if not v.isdigit():
bad['value_not_int'] = bad.get('value_not_int', 0) + 1
continue
n = int(v)
if n <= 0 or n > MAX_COUNT:
bad['value_out_of_range'] = bad.get('value_out_of_range', 0) + 1
continue
code = (r.get('major_code') or '').strip()
if code and not CODE_OK.match(code):
bad['code_format'] = bad.get('code_format', 0) + 1
r = dict(r, data_flag='code_format_odd')
if not code and not (r.get('major_name') or '').strip():
bad['empty_major_key'] = bad.get('empty_major_key', 0) + 1
continue
key = tuple((r.get(k) or '') for k in keys)
if key in seen:
bad['dup_business_key'] = bad.get('dup_business_key', 0) + 1
continue
seen.add(key)
out.append(r)
return out, bad
def ingest(rows, table, tag, reject):
z = sqlite3.connect(ZX, timeout=300)
z.execute(DDL_TM if table == 'y2027_tuimian_2027' else DDL_PLAN)
cols = [d[0] for d in z.execute('select * from "%s" limit 0' % table).description
if d[0] != '_ingested_at']
now = time.strftime('%Y-%m-%dT%H:%M:%S')
sql = 'INSERT OR IGNORE INTO "%s" ("%s","_ingested_at") VALUES (%s,?)' % (
table, '","'.join(cols), ','.join(['?'] * len(cols)))
n = 0
for r in rows:
rec = []
for c in cols:
v = r.get(c, '')
if v is None:
v = ''
rec.append(str(v) if not isinstance(v, str) else v)
cur = z.execute(sql, rec + [now])
n += cur.rowcount
z.commit()
z.execute('INSERT INTO changelog ("ts","script","object","action","n_rows","detail") '
'VALUES (?,?,?,?,?,?)',
(time.strftime('%Y-%m-%d %H:%M:%S'), 'round2_roster_20261101', table,
'insert', n,
'basis=%s cand=%d reject=%s' % (tag, len(rows),
json.dumps(reject, ensure_ascii=False))))
z.commit()
tot = z.execute('select count(*) from "%s"' % table).fetchone()[0]
print('%s: 新增 %d / 候选 %d 表内合计 %d 拒绝 %s' % (table, n, len(rows), tot, reject))
return z, n, tot
if __name__ == '__main__':
src = sys.argv[1]
data = json.load(open(src, encoding='utf-8'))
tm = [x for x in data if x.get('target') == 'tuimian']
pl = [x for x in data if x.get('target') == 'plan']
K_TM = ('school', 'year', 'college', 'college_code', 'major_code', 'major_name',
'kind', 'basis', 'plan_type')
K_PL = ('school', 'year', 'college', 'college_code', 'major_code', 'major_name',
'study_mode', 'kind')
tm, rj1 = validate(tm, K_TM)
pl, rj2 = validate(pl, K_PL)
print('候选 tuimian=%d plan=%d' % (len(tm), len(pl)))
if rj1:
print(' tuimian 拒绝:', rj1)
if rj2:
print(' plan 拒绝:', rj2)
z = None
if tm:
z, n1, t1 = ingest(tm, 'y2027_tuimian_2027', 'tuimian', rj1)
if pl:
zz, n2, t2 = ingest(pl, 'y2027_plan_2027', 'plan', rj2)
if z is None:
z = zz
if z:
z.close()
@@ -0,0 +1,265 @@
# -*- coding: utf-8 -*-
"""从 grid 缓存解析名单:严格表头判定 + 内容定位兜底。
上一版把每行都判成表头(数据行的学院值本身含「学院」二字),导致整表吞光。
现在:
1. 表头必须出现「姓名/考生编号/证件号码/身份证号/录取数/可用计划」这类只可能在表头出现的词,
且该行不含 6 位专业代码。
2. 表头能定列就按表头;定不到(如东北大学「代码及名称」合并表头、实际拆成两列)就按内容定位:
6 位代码左侧最近的中文短格=学院,右侧最近的中文=专业名。
3. 院系列是合并单元格,空值继承上一行。
4. 「汇总/合计/小计」行先排除再分组。
"""
import glob
import json
import os
import re
import sys
CACHE = r'D:/择校/运行暂存/agent_ingest_20261031/grid_cache'
SCHOOL = {'bnu': '北京师范大学', 'bucm': '北京中医药大学', 'cumtb': '中国矿业大学(北京)',
'ecnu': '华东师范大学', 'neu': '东北大学', 'njust': '南京理工大学',
'nwpu': '西北工业大学', 'scut': '华南理工大学', 'seu': '东南大学',
'shnu': '上海师范大学'}
SCHOOL_ALIAS = {'华东师大': '华东师范大学'}
TOTAL_WORDS = ('汇总', '合计', '小计', '共计', '总计')
HDR_WORDS = re.compile(r'姓名|考生编号|证件号码|身份证号|录取数|可用计划|接收人数|拟接收|综合考核|复试成绩')
CODE_RE = re.compile(r'^\d{6}$|^\d{4}[A-Z]\d{1,2}$|^\d{2}[A-Z]\d{3,4}$')
ANYCODE = re.compile(r'(?<![\dA-Za-z])(\d{6}|[0-9]{4}[A-Z][0-9]{1,2}|[0-9]{2}[A-Z][0-9]{3,4})(?![\dA-Za-z])')
COLLEGE_TAIL = re.compile(r'(学院|系|研究所|研究中心|研究院|中心|部|所|基地|实验室|书院)$')
def clean(s):
"""规范化空白,并把 PDF 抽出来的 Kangxi 兼容部首(⼈/⼯/⽣)折回正规汉字。
全角标点原样保留:校名归一只该去空白与修字形,不该把「()」变成「()」。
"""
t = re.sub(r'\s+', ' ', str(s or '')).strip()
if not t:
return t
out = []
for ch in t:
if 0xFF01 <= ord(ch) <= 0xFF5E or 0x3000 <= ord(ch) <= 0x303F:
out.append(ch)
else:
out.append(_nfkc(ch))
return ''.join(out)
def _nfkc(ch):
import unicodedata
o = unicodedata.normalize('NFKC', ch)
return o if len(o) == 1 and 0x4E00 <= ord(o) <= 0x9FFF else ch
def header_map(cells):
m = {}
for i, raw in enumerate(cells):
c = clean(raw).replace(' ', '')
if not c:
continue
if 'code' not in m and '专业代码' in c:
m['code'] = i
elif 'name' not in m and re.search(r'(专业|学科)(名称|全称|及名称)$|^(拟录取|录取)?(专业|学科)$|^学科专业$', c):
m['name'] = i
elif 'college_code' not in m and re.search(r'(学院|院系所?|单位)(代码|码)$', c):
m['college_code'] = i
elif 'college' not in m and (re.search(r'^(拟录取|录取)?(学院|院系|院系所|单位)$', c)
or (re.search(r'(学院|院系|单位|研究院)', c) and re.search(r'(名称|全称)', c))):
m['college'] = i
elif 'count' not in m and re.search(r'^(录取数|接收人数|拟接收|统考可用计划|可用计划|人数)$', c):
m['count'] = i
elif 'kind' not in m and re.search(r'^(招生)?类型$|^层次$|^类型$', c):
m['kind'] = i
elif 'special' not in m and re.search(r'^专项计划(名称)?$', c):
m['special'] = i
elif 'mode' not in m and re.search(r'学习方式$|^形式$', c):
m['mode'] = i
return m
def is_header(cells):
j = ''.join(clean(c) for c in cells)
if not HDR_WORDS.search(j):
return False
if ANYCODE.search(j):
return False
return True
def locate_by_content(cells):
"""表头不可用时按内容定:6 位代码 + 左邻学院 + 右邻专业名。"""
ci = None
for i, c in enumerate(cells):
cc = c.replace(' ', '')
if CODE_RE.match(cc):
ci = i
break
if ci is None:
return None
college = name = ''
for j in range(ci - 1, -1, -1):
v = cells[j]
if v and COLLEGE_TAIL.search(v) and len(v) <= 30:
college = v
break
for j in range(ci + 1, len(cells)):
v = cells[j]
if v and not CODE_RE.match(v.replace(' ', '')) and re.search(r'[一-龥]', v) \
and not re.search(r'不区分研究方向|研究方向', v):
name = v
break
return college, name
def parse(path, year, url_index):
d = json.load(open(path, encoding='utf-8'))
key = d['file'].split('_')[0]
school = SCHOOL_ALIAS.get(SCHOOL.get(key, key), SCHOOL.get(key, key))
stem = d['file'].rsplit('.', 1)[0]
url = ''
best = -1
for t, v in url_index.items():
if stem.startswith(t + '_') or stem == t:
if len(t) > best:
best, url = len(t), v['url']
hm, rows = None, []
last_college = ''
count_label = ''
n_hdr = n_skip_total = n_body = n_content = n_special = 0
for pages in d['pages']:
for raw in pages:
cells = [clean(c) for c in raw]
if not any(cells):
continue
if is_header(cells):
cand = header_map(cells)
if cand:
hm, n_hdr = cand, n_hdr + 1
count_label = (cells[cand['count']] if 'count' in cand
and cand['count'] < len(cells) else '')
last_college = ''
continue
if any(w in ''.join(cells) for w in TOTAL_WORDS):
n_skip_total += 1
continue
code = name = college = kind = mode = count = college_code = ''
special = ''
if hm:
def at(k):
i = hm.get(k)
return cells[i] if (i is not None and i < len(cells)) else ''
college_code = at('college_code')
special = at('special')
v = at('code')
m = ANYCODE.search(v.replace(' ', '')) if v else None
code = m.group(1) if m else ''
name = at('name')
if m and not name and code and len(v.replace(' ', '')) > len(code):
tail = v.replace(' ', '').split(code, 1)[-1] # 「080200机械工程」连写
name = re.sub(r'^[一-龥()()·A-Za-z0-9]{0,4}?(?=[一-龥])', '', tail)[:40] if re.search(r'[一-龥]', tail) else ''
name = tail if re.search(r'[一-龥]', tail) else ''
kind = at('kind')
mode = at('mode')
count = at('count')
college = at('college') or last_college or college_code
if code or name:
if (not name) or (not college) or college.isdigit():
lc = locate_by_content(cells) # 「码及名称」合并表头会拆列
if lc:
if college.isdigit() and len(college) <= 4:
college_code = college_code or college
if not college or college.isdigit():
college = lc[0] or college
name = name or lc[1]
n_content += 1
elif not code and not name:
lc = locate_by_content(cells)
if lc:
college = college or lc[0]
name = lc[1]
n_content += 1
if not code:
for v in cells:
m = ANYCODE.search(v.replace(' ', ''))
if m and v.replace(' ', '') != name:
code = m.group(1)
break
else:
lc = locate_by_content(cells)
if not lc:
continue
college, name = lc[0] or last_college, lc[1]
for v in cells:
m = ANYCODE.search(v.replace(' ', ''))
if m:
code = m.group(1)
break
n_content += 1
if special and not re.search(r'^0$|普通计划|无|^-$', special.replace(' ', '')):
n_special += 1
continue # 专项计划不计入推免接收人数,否则合计虚高
if college.isdigit() and len(college) <= 4:
college_code = college_code or college
college = '' # 代码列被当成名称列:搬回 college_code
if college:
last_college = college
if not (code or name):
continue
rows.append(dict(school=school, year=year, college=college,
college_code=college_code, major_code=code,
major_name=name, kind=kind, study_mode=mode, value=count,
source_url=url, local_file=d['file'],
_count_label=count_label))
n_body += 1
diag = '%-10s %-16s hdr=%-3d 总行=%-5d 汇总行=%-3d 专项行=%-4d 内容兜底=%-4d 无表头=%s' % (
key, school, n_hdr, n_body, n_skip_total, n_special, n_content,
'yes' if hm is None else 'no')
return school, rows, diag
def group(school, rows):
"""直接计数行原样保留;名单行按业务键分组。"""
direct, agg, order = [], {}, []
for r in rows:
if r['value'] and re.match(r'^\d{1,5}$', r['value']):
rr = dict(r, value=r['value'], basis='count_direct')
rr['target'] = 'plan' if '计划' in (r.get('_count_label') or '') else 'tuimian'
direct.append(rr)
continue
k = (r['school'], r['year'], r['college'], r['college_code'], r['major_code'],
r['major_name'], r['kind'], r['study_mode'])
if k not in agg:
agg[k] = dict(r, value='1', basis='roster_derived')
order.append(k)
else:
agg[k]['value'] = str(int(agg[k]['value']) + 1)
for r in agg.values():
r['target'] = 'tuimian'
return direct, [agg[k] for k in order]
def main():
tag = sys.argv[1] if len(sys.argv) > 1 else 'tm27'
year = '2027'
url_index = json.load(open(r'D:/择校/运行暂存/agent_ingest_20261031/round2_tag_index.json',
encoding='utf-8'))
out, diag = [], []
for p in sorted(glob.glob(os.path.join(CACHE, tag + '_*.json'))):
try:
school, rows, d = parse(p, year, url_index)
except Exception as e:
diag.append('%s FAIL %s' % (os.path.basename(p), e))
continue
direct, derived = group(school, rows)
diag.append(d + ' 直接计数=%d 分组=%d' % (len(direct), len(derived)))
out += direct + derived
dst = r'D:/择校/运行暂存/agent_ingest_20261031/cand_round2_pdf.json'
with open(dst, 'w', encoding='utf-8') as f:
json.dump(out, f, ensure_ascii=False, indent=1)
print('\n'.join(diag))
print('rows=%d -> %s' % (len(out), dst))
if __name__ == '__main__':
main()