How the data is built

The questions come from Taiwan's published past examination papers. This page records what happens in between — because the real risk in a benchmark is not a few missing questions, it is an answer key that lands one question to the left and nobody notices.

From a paper to a question

Fetch the Ministry of Examination's open-data paper index; keep the multiple-choice papers that publish an answer key.
Download the question and answer-key PDFs, rate-limited and cached on disk, so a rerun costs the service nothing.
Extract the text layer with PyMuPDF — these are born-digital PDFs, no OCR involved.
Filter, deduplicate, and write out in the Twinkle Eval MCQ format.

Parsing hazards

None of these crash anything. They just corrupt the data quietly, so each one is recorded.

Hazard	Handling
Option markers are Private Use Area glyphs (U+E18C–U+E190), not the letters A–D	Mapped back to letters; older layouts use `A.` or `(A)` instead
Answer keys are grids whose text layer comes out shuffled — question 13 can appear after question 40	Cells are paired by page coordinates, with the tolerance derived from the grid pitch. Never by reading order
The first answer cell of a row is glued to the row label (`答案Ｃ`)	Left unhandled that cell vanishes and the whole row shifts one column right — every answer on the paper wrong
A bare number inside a stem looks like the next question number	Question numbers must run 1..N monotonically, or the line is treated as body text
Options indented with U+00A0 / U+3000, which NFC leaves alone	Those spaces are folded explicitly. NFKC would fold them too, but it also rewrites 「，」 to an ASCII comma
CJK Compatibility Ideographs mixed into the text	Normalised with NFC, not NFKC

Every paper is cross-checked against the question count its own answer sheet declares. A mismatch rejects the paper rather than importing a key that has already shifted.

What is excluded, and why

Each rule below was added after measurement, and several were loosened after measurement showed they were discarding good questions. That direction of error is the dangerous one: it is invisible in the output, and it makes the benchmark quietly smaller and less representative.

Excluded	Rule
Multi-select and voided items	The answer sheet marks them
English-language items	Only when the stem contains no Chinese at all. Judging the options too would discard the many Chinese questions answered in English terminology — 下列何者可以抑制 ribonucleotide reductase 的活性？ with options like `dATP` is a Chinese question
Items needing a figure or table	Requires a deictic reference (as shown below, in Figure 1, according to the table). Matching 圖 loosely would discard 圖書館 (library), 圖書分類 (classification), 意圖 (intent)
Follow-up items in a question group	They open with 「承上題」 and refer to the previous question's scenario. Lifted out of the paper nobody can answer them. The setup question is self-contained and stays
Items whose option swallowed adjacent text	One option far outside the 99.9th percentile (95 characters) and several times longer than its siblings. Imbalance is required — four long options is just a long question
Questions already in tw-legal-benchmark-v2	Matched item by item, not by dropping subjects: Taiwan's statutory subjects are examined inside otherwise non-legal categories
Republished items with conflicting keys	When the official versions disagree the item is dropped, not guessed

Deduplication

The same paper is republished under several examination categories. Items are fingerprinted on the stem plus the sorted option set, so a reprint with the options in a different order still matches.

The scale is not marginal. Chinese Language is a subject every category sits: 35,334 questions before deduplication, 4,191 after — an 8× inflation, with one question appearing in 22 different papers. Those 22 independently published copies came out of the parser with identical answer keys, which is the strongest available evidence that the parsing is correct.

Checks before publishing

Row contract: question, A–D, answer all non-empty strings, the answer among the options.
Answer-position skew — any letter holding more than 40% fails. This is the loudest symptom of a shifted key.
Residual duplicates, surviving English items, merged stems.
benchmark.csv mirrors benchmark.jsonl field by field. The Hub card points load_dataset at the CSV while every other check reads the JSONL, and 2,645 items carry a quote, comma or newline — a quoting bug there would corrupt what people download with every other check still green.

Anything that fails, push_to_hub refuses to publish.

Known properties

Contamination risk. These are public past papers, widely discussed online and very likely in pretraining corpora. Treat absolute scores with caution; the suite is built for relative comparison between models.
Answer-position skew exists in the source papers. Shuffle the options when evaluating.
Item difficulty is set by the examination committee, not by us.

資料如何建立

題目來自考選部公開的考畢試題。這一頁記錄的是中間發生了什麼 —— 因為評測集真正的風險不是少了幾題，而是答案悄悄錯位一格而沒有人發現。

一份試卷怎麼變成題目

抓取考選部公開的試題索引，篩出有標準答案的測驗題試卷。
下載試題與答案 PDF，限速並落地快取，重跑不會再打伺服器。
以 PyMuPDF 取出文字層 —— PDF 是電子文件，不需 OCR。
過濾、去重，輸出成 Twinkle Eval MCQ 格式。

解析陷阱

這些都不會讓程式崩潰，只會讓資料靜靜地壞掉，所以逐項記錄。

陷阱	處理
選項標記是造字區私用碼（U+E18C–U+E190），不是字母 A–D	轉回字母；舊版排版另有 `A.` 與 `(A)` 兩種形態
答案 PDF 是表格，文字層抽出來順序是亂的 —— 第 13 題可能排在第 40 題之後	改用座標配對，容差由格線間距推導，絕不依閱讀順序
每列第一格答案黏在「答案」標籤上（`答案Ｃ`）	不處理的話該格會消失，整列往右錯一位 —— 全份答案皆錯
題幹中獨立成行的數字被誤認為新題號	題號必須 1..N 連續遞增，否則視為內文
選項用 U+00A0 / U+3000 縮排，NFC 不會處理	單獨折疊這些空白字元；NFKC 雖可處理，但也會把「，」改成半形逗號
文中混雜相容漢字	以 NFC 正規化，不用 NFKC

每份試卷都會跟答案卷自己宣告的題數交叉驗證。對不上就整份退掉，而不是匯入一份已經錯位的答案。

哪些題目被排除，為什麼

下列每一條都是量測後才加上的，而且其中好幾條在量測後放寬過。過度過濾是比較危險的方向 —— 它在輸出裡看不見，只會讓評測集悄悄變小、變得不具代表性。

排除	判準
複選題、廢題（一律給分）	答案卷本身標示
英文題	只有題幹完全沒有中文字才算。若把選項也算進去，下列何者可以抑制 ribonucleotide reductase 的活性？配 `dATP` 這種醫藥考題會被誤刪 —— 那是中文題
需要看圖表的題目	必須是指示性引用（如下圖所示、圖（一）中、根據下表）。寬鬆比對「圖」會連圖書館、圖書分類、意圖一起砍掉
題組的後續題	開頭寫「承上題」，情境在前一題。抽出試卷後任何人都答不出來。設定題本身自足，保留
選項吞掉相鄰文字	某選項遠超第 99.9 百分位（95 字）且比同題其他選項長數倍。必須失衡 —— 四個選項都長就只是題目長
已收錄於 tw-legal-benchmark-v2 的題目	逐題比對，不是整科排除：台灣的法規科目是夾在非法律類科裡考的
答案鑰匙互相矛盾的重刊題	官方版本不一致時整題丟棄，不猜

去重

同一份試卷會在多個類科重複刊登。題目以「題幹 + 排序後的選項集合」建立指紋，因此重刊時選項順序被重排也照樣比對得到。

規模不容忽視。國文是所有類科都要考的共同科目：去重前 35,334 題，去重後 4,191 題 —— 8 倍膨脹，最高一題出現在 22 份不同試卷。而這 22 個獨立刊登版本經解析後答案完全一致，這是解析正確性最強的旁證。

發佈前的檢查

欄位契約：question、A–D、answer 皆為非空字串，答案在選項內。
答案位置偏斜 —— 任一字母佔比超過 40% 即擋下。這是答案錯位最明顯的症狀。
殘留重複、漏網英文題、合併題幹。
benchmark.csv 與 benchmark.jsonl 逐欄一致。 Hub 的 load_dataset 讀的是 CSV，而其他檢查都跑在 JSONL 上，且有 2,645 題含引號／逗號／換行 —— 跳脫一旦出錯，下載到的資料會壞掉而所有檢查依然全綠。

任何一項不過，push_to_hub 直接拒絕發佈。

已知性質

污染風險。這些是公開考古題，網路上廣泛討論，極可能已在預訓練語料中。絕對分數請保留懷疑，本套件適合用於模型之間的相對比較。
答案位置偏斜存在於原始考卷，評測時請開啟選項洗牌。
難度由典試委員決定，不是我們。