https://opentwbench.ai/methodology.html

1<!doctype html>
2<html lang="en">
3<head>
4<meta charset="utf-8">
5<meta name="viewport" content="width=device-width, initial-scale=1">
6<title>Methodology — OpenTWBench</title>
7<meta name="description" content="How OpenTWBench turns Taiwan's published examination papers into benchmarks: parsing hazards, exclusion rules and quality checks.">
8<meta property="og:title" content="Methodology — OpenTWBench">
9<meta property="og:description" content="How OpenTWBench scores models — boxed-answer extraction, deterministic option shuffling, full-precision serving, and a parse rate beside every score — built for reproducible, comparable results.">
10<meta property="og:type" content="website">
11<meta property="og:image" content="https://opentwbench.ai/assets/og-image.png">
12<meta property="og:image:width" content="1200">
13<meta property="og:image:height" content="630">
14<meta name="twitter:card" content="summary_large_image">
15<meta name="twitter:image" content="https://opentwbench.ai/assets/og-image.png">
16<link rel="icon" href="assets/favicon.svg?v=5019eba3f4" type="image/svg+xml">
17<link rel="icon" href="assets/favicon.ico?v=e7d4978bfa" sizes="any">
18<link rel="apple-touch-icon" href="assets/apple-touch-icon.png?v=62d5c134f9">
19<link rel="mask-icon" href="assets/safari-pinned-tab.svg?v=164f29ae60" color="#6676FF">
20<link rel="manifest" href="assets/site.webmanifest?v=2ca761e1b6">
21<link rel="preconnect" href="https://fonts.googleapis.com">
22<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
23<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter+Tight:[email protected]&family=Playfair+Display:[email protected]&display=swap">
24<link rel="stylesheet" href="assets/styles.css?v=8ed7de4708">
25</head>
26<body>
27<header class="site-header">
28  <div class="wrap header-inner">
29    <a class="brand" href="/">
30      <span class="brand-wordmark" role="img" aria-label="OpenTWBench"></span>
31    </a>
32    <nav class="site-nav" id="site-nav">
33      <a href="/#leaderboard" data-i18n="nav.leaderboard">Leaderboard</a>
34      <a href="/benchmarks.html">Benchmarks</a>
35      <a href="/subjects.html" data-i18n="nav.subjects">Subjects</a>
36      <a href="/methodology.html" data-i18n="nav.methodology">Methodology</a>
37      <a href="/#cite" data-i18n="nav.cite">Cite</a>
38      <a class="nav-mark" href="https://huggingface.co/OpenTWBench" rel="noopener"><img src="assets/logos/huggingface.svg?v=fb54b5823d" alt="" width="18" height="18"><span data-i18n="nav.hf">Hugging Face</span></a>
39    </nav>
40    <div class="header-tools">
41      <button id="theme-toggle" class="icon-button" type="button"
42              data-i18n-attr="aria-label: theme.toggle">◐</button>
43      <button id="nav-toggle" class="icon-button nav-toggle" type="button"
44              data-i18n-attr="aria-label: nav.menu" aria-expanded="false">☰</button>
45    </div>
46  </div>
47</header>
48
49<main class="wrap prose">
50  <div class="only-en">
51  <h1>How the data is built</h1>
52  <p class="lede">
53    The questions come from Taiwan's published past examination papers. This page records
54    what happens in between — because the real risk in a benchmark is not a few missing
55    questions, it is <strong>an answer key that lands one question to the left</strong> and
56    nobody notices.
57  </p>
58
59  <h2>From a paper to a question</h2>
60  <ol class="steps">
61    <li>Fetch the Ministry of Examination's open-data paper index; keep the multiple-choice papers that publish an answer key.</li>
62    <li>Download the question and answer-key PDFs, rate-limited and cached on disk, so a rerun costs the service nothing.</li>
63    <li>Extract the text layer with PyMuPDF — these are born-digital PDFs, no OCR involved.</li>
64    <li>Filter, deduplicate, and write out in the Twinkle Eval MCQ format.</li>
65  </ol>
66
67  <h2>Parsing hazards</h2>
68  <p>None of these crash anything. They just corrupt the data quietly, so each one is recorded.</p>
69  <div class="table-wrap">
70  <table class="doc-table">
71    <thead><tr><th>Hazard</th><th>Handling</th></tr></thead>
72    <tbody>
73      <tr><td>Option markers are <strong>Private Use Area glyphs</strong> (U+E18C–U+E190), not the letters A–D</td>
74          <td>Mapped back to letters; older layouts use <code>A.</code> or <code>(A)</code> instead</td></tr>
75      <tr><td>Answer keys are grids whose <strong>text layer comes out shuffled</strong> — question 13 can appear after question 40</td>
76          <td>Cells are paired <strong>by page coordinates</strong>, with the tolerance derived from the grid pitch. Never by reading order</td></tr>
77      <tr><td>The first answer cell of a row is <strong>glued to the row label</strong> (<code>答案Ｃ</code>)</td>
78          <td>Left unhandled that cell vanishes and the whole row shifts one column right — every answer on the paper wrong</td></tr>
79      <tr><td>A bare number inside a stem looks like the next question number</td>
80          <td>Question numbers must run <strong>1..N monotonically</strong>, or the line is treated as body text</td></tr>
81      <tr><td>Options indented with <strong>U+00A0 / U+3000</strong>, which NFC leaves alone</td>
82          <td>Those spaces are folded explicitly. NFKC would fold them too, but it also rewrites 「，」 to an ASCII comma</td></tr>
83      <tr><td>CJK Compatibility Ideographs mixed into the text</td>
84          <td>Normalised with <strong>NFC</strong>, not NFKC</td></tr>
85    </tbody>
86  </table>
87  </div>
88  <p class="note">
89    Every paper is cross-checked against <strong>the question count its own answer sheet declares</strong>.

90    A mismatch rejects the paper rather than importing a key that has already shifted.
91  </p>
92
93  <h2>What is excluded, and why</h2>
94  <p>
95    Each rule below was added after measurement, and several were <strong>loosened</strong> after
96    measurement showed they were discarding good questions. That direction of error is the
97    dangerous one: it is invisible in the output, and it makes the benchmark quietly smaller
98    and less representative.
99  </p>
100  <div class="table-wrap">
101  <table class="doc-table">
102    <thead><tr><th>Excluded</th><th>Rule</th></tr></thead>
103    <tbody>
104      <tr><td>Multi-select and voided items</td><td>The answer sheet marks them</td></tr>
105      <tr><td>English-language items</td>
106          <td>Only when the <strong>stem contains no Chinese at all</strong>. Judging the options too would discard the many Chinese questions answered in English terminology — <em>下列何者可以抑制 ribonucleotide reductase 的活性？</em> with options like <code>dATP</code> is a Chinese question</td></tr>
107      <tr><td>Items needing a figure or table</td>
108          <td>Requires a <strong>deictic reference</strong> (<em>as shown below</em>, <em>in Figure 1</em>, <em>according to the table</em>). Matching 圖 loosely would discard 圖書館 (library), 圖書分類 (classification), 意圖 (intent)</td></tr>
109      <tr><td>Follow-up items in a question group</td>
110          <td>They open with 「承上題」 and refer to the previous question's scenario. Lifted out of the paper <strong>nobody can answer them</strong>. The setup question is self-contained and stays</td></tr>
111      <tr><td>Items whose option swallowed adjacent text</td>
112          <td>One option far outside the 99.9th percentile (95 characters) <strong>and</strong> several times longer than its siblings. Imbalance is required — four long options is just a long question</td></tr>
113      <tr><td>Questions already in tw-legal-benchmark-v2</td>
114          <td>Matched <strong>item by item</strong>, not by dropping subjects: Taiwan's statutory subjects are examined inside otherwise non-legal categories</td></tr>
115      <tr><td>Republished items with conflicting keys</td>
116          <td>When the official versions disagree the item is <strong>dropped, not guessed</strong></td></tr>
117    </tbody>
118  </table>
119  </div>
120
121  <h2>Deduplication</h2>
122  <p>
123    The same paper is republished under several examination categories. Items are fingerprinted
124    on the stem plus the <em>sorted</em> option set, so a reprint with the options in a different
125    order still matches.
126  </p>
127  <p>
128    The scale is not marginal. <strong>Chinese Language</strong> is a subject every category sits:
129    35,334 questions before deduplication, <strong>4,191</strong> after — an <strong>8×</strong>
130    inflation, with one question appearing in <strong>22</strong> different papers. Those 22
131    independently published copies came out of the parser with <strong>identical answer keys</strong>,
132    which is the strongest available evidence that the parsing is correct.
133  </p>
134
135  <h2>Checks before publishing</h2>
136  <ul>
137    <li>Row contract: <code>question</code>, <code>A</code>–<code>D</code>, <code>answer</code> all non-empty strings, the answer among the options.</li>
138    <li><strong>Answer-position skew</strong> — any letter holding more than 40% fails. This is the loudest symptom of a shifted key.</li>
139    <li>Residual duplicates, surviving English items, merged stems.</li>
140    <li><strong><code>benchmark.csv</code> mirrors <code>benchmark.jsonl</code> field by field.</strong> The Hub card points <code>load_dataset</code> at the CSV while every other check reads the JSONL, and 2,645 items carry a quote, comma or newline — a quoting bug there would corrupt what people download with every other check still green.</li>
141  </ul>
142  <p class="note">Anything that fails, <code>push_to_hub</code> refuses to publish.</p>
143
144  <h2>Known properties</h2>
145  <ul>
146    <li><strong>Contamination risk.</strong> These are public past papers, widely discussed online and very likely in pretraining corpora. Treat absolute scores with caution; the suite is built for <em>relative</em> comparison between models.</li>
147    <li><strong>Answer-position skew</strong> exists in the source papers. Shuffle the options when evaluating.</li>
148    <li><strong>Item difficulty is set by the examination committee</strong>, not by us.</li>
149  </ul>
150</div>
151  <div class="only-zh">
152  <h1>資料如何建立</h1>
153  <p class="lede">
154    題目來自考選部公開的考畢試題。這一頁記錄的是中間發生了什麼 ——
155    因為評測集真正的風險不是少了幾題，而是<strong>答案悄悄錯位一格</strong>而沒有人發現。
156  </p>
157
158  <h2>一份試卷怎麼變成題目</h2>
159  <ol class="steps">
160    <li>抓取考選部公開的試題索引，篩出有標準答案的測驗題試卷。</li>
161    <li>下載試題與答案 PDF，限速並落地快取，重跑不會再打伺服器。</li>
162    <li>以 PyMuPDF 取出文字層 —— PDF 是電子文件，不需 OCR。</li>
163    <li>過濾、去重，輸出成 Twinkle Eval MCQ 格式。</li>
164  </ol>
165
166  <h2>解析陷阱</h2>
167  <p>

167這些都不會讓程式崩潰，只會讓資料靜靜地壞掉，所以逐項記錄。</p>
168  <div class="table-wrap">
169  <table class="doc-table">
170    <thead><tr><th>陷阱</th><th>處理</th></tr></thead>
171    <tbody>
172      <tr><td>選項標記是<strong>造字區私用碼</strong>（U+E18C–U+E190），不是字母 A–D</td>
173          <td>轉回字母；舊版排版另有 <code>A.</code> 與 <code>(A)</code> 兩種形態</td></tr>
174      <tr><td>答案 PDF 是表格，<strong>文字層抽出來順序是亂的</strong> —— 第 13 題可能排在第 40 題之後</td>
175          <td>改用<strong>座標配對</strong>，容差由格線間距推導，絕不依閱讀順序</td></tr>
176      <tr><td>每列第一格答案<strong>黏在「答案」標籤上</strong>（<code>答案Ｃ</code>）</td>
177          <td>不處理的話該格會消失，整列往右錯一位 —— 全份答案皆錯</td></tr>
178      <tr><td>題幹中獨立成行的數字被誤認為新題號</td>
179          <td>題號必須 <strong>1..N 連續遞增</strong>，否則視為內文</td></tr>
180      <tr><td>選項用 <strong>U+00A0 / U+3000</strong> 縮排，NFC 不會處理</td>
181          <td>單獨折疊這些空白字元；NFKC 雖可處理，但也會把「，」改成半形逗號</td></tr>
182      <tr><td>文中混雜相容漢字</td>
183          <td>以 <strong>NFC</strong> 正規化，不用 NFKC</td></tr>
184    </tbody>
185  </table>
186  </div>
187  <p class="note">
188    每份試卷都會跟<strong>答案卷自己宣告的題數</strong>交叉驗證。對不上就整份退掉，
189    而不是匯入一份已經錯位的答案。
190  </p>
191
192  <h2>哪些題目被排除，為什麼</h2>
193  <p>
194    下列每一條都是量測後才加上的，而且其中好幾條在量測後<strong>放寬過</strong>。
195    過度過濾是比較危險的方向 —— 它在輸出裡看不見，只會讓評測集悄悄變小、變得不具代表性。
196  </p>
197  <div class="table-wrap">
198  <table class="doc-table">
199    <thead><tr><th>排除</th><th>判準</th></tr></thead>
200    <tbody>
201      <tr><td>複選題、廢題（一律給分）</td><td>答案卷本身標示</td></tr>
202      <tr><td>英文題</td>
203          <td>只有<strong>題幹完全沒有中文字</strong>才算。若把選項也算進去，
204              <em>下列何者可以抑制 ribonucleotide reductase 的活性？</em> 配 <code>dATP</code>
205              這種醫藥考題會被誤刪 —— 那是中文題</td></tr>
206      <tr><td>需要看圖表的題目</td>
207          <td>必須是<strong>指示性引用</strong>（如下圖所示、圖（一）中、根據下表）。
208              寬鬆比對「圖」會連<strong>圖書</strong>館、<strong>圖書</strong>分類、意<strong>圖</strong>一起砍掉</td></tr>
209      <tr><td>題組的後續題</td>
210          <td>開頭寫「承上題」，情境在前一題。抽出試卷後<strong>任何人都答不出來</strong>。設定題本身自足，保留</td></tr>
211      <tr><td>選項吞掉相鄰文字</td>
212          <td>某選項遠超第 99.9 百分位（95 字）<strong>且</strong>比同題其他選項長數倍。
213              必須失衡 —— 四個選項都長就只是題目長</td></tr>
214      <tr><td>已收錄於 tw-legal-benchmark-v2 的題目</td>
215          <td><strong>逐題</strong>比對，不是整科排除：台灣的法規科目是夾在非法律類科裡考的</td></tr>
216      <tr><td>答案鑰匙互相矛盾的重刊題</td>
217          <td>官方版本不一致時<strong>整題丟棄，不猜</strong></td></tr>
218    </tbody>
219  </table>
220  </div>
221
222  <h2>去重</h2>
223  <p>
224    同一份試卷會在多個類科重複刊登。題目以「題幹 + <em>排序後</em>的選項集合」建立指紋，
225    因此重刊時選項順序被重排也照樣比對得到。
226  </p>
227  <p>
228    規模不容忽視。<strong>國文</strong>是所有類科都要考的共同科目：去重前 35,334 題，
229    去重後 <strong>4,191</strong> 題 —— <strong>8 倍</strong>膨脹，最高一題出現在
230    <strong>22</strong> 份不同試卷。而這 22 個獨立刊登版本經解析後<strong>答案完全一致</strong>，
231    這是解析正確性最強的旁證。
232  </p>
233
234  <h2>發佈前的檢查</h2>
235  <ul>
236    <li>欄位契約：<code>question</code>、<code>A</code>–<code>D</code>、<code>answer</code> 皆為非空字串，答案在選項內。</li>
237    <li><strong>答案位置偏斜</strong> —— 任一字母佔比超過 40% 即擋下。這是答案錯位最明顯的症狀。</li>
238    <li>殘留重複、漏網英文題、合併題幹。</li>
239    <li><strong><code>benchmark.csv</code> 與 <code>benchmark.jsonl</code> 逐欄一致。</strong>
240        Hub 的 <code>load_dataset</code> 讀的是 CSV，而其他檢查都跑在 JSONL 上，
241        且有 2,645 題含引號／逗號／換行 —— 跳脫一旦出錯，下載到的資料會壞掉而所有檢查依然全綠。</li>
242  </ul>
243  <p class="note">任何一項不過，<code>push_to_hub</code> 直接拒絕發佈。</p>
244
245  <h2>已知性質</h2>
246  <ul>
247    <li><strong>污染風險。</strong>這些是公開考古題，網路上廣泛討論，極可能已在預訓練語料中。
248        絕對分數請保留懷疑，本套件適合用於模型之間的<em>相對</em>比較。</li>
249    <li><strong>答案位置偏斜</strong>存在於原始考卷，評測時請開啟選項洗牌。</li>
250    <li><strong>難度由典試委員決定</strong>，不是我們。</li>
251  </ul>
252</div>
253</main>
254<footer class="site-footer">
255  <div class="wrap footer-inner">
256    <p data-i18n="footer.rights">Each benchmark states its own source and licence on its dataset card — examination questions are Ministry of Examination (考選部) open data; other benchmarks derive from their respective corpora. Packaging and metadata: Apache-2.0.</p>
257    <div class="footer-links">
258      <a href="/subjects.html" data-i18n="nav.subjects">Subjects</a>
259      <a href="/methodology.html" data-i18n="nav.methodology">Methodology</a>
260      <a href="https://huggingface.co/OpenTWBench" rel="noopener">Hugging Face</a>
261      <a href="https://github.com/ai-twinkle/Eval" rel="noopener">Twinkle Eval</a>
262    </div>
263  </div>
264</footer>

265<script src="assets/i18n.js?v=d947cd2fba"></script>

265

266<script src="assets/app.js?v=9d81892857"></script>

266
267
268</body>
269</html>