PageSourceSearch

https://paracrawl.eu/moredata

html paracrawl.eu collected 2026-10-02 03:42:16 UTC 35,821 bytes, 666 lines download raw bytes

1
2<!doctype html>
3<html prefix="og: http://ogp.me/ns#" lang="en-gb" dir="ltr">
4    <head>
5        <meta charset="utf-8">
6        <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
7        <link rel="canonical" href="https://paracrawl.eu/moredata">
8        <base href="https://paracrawl.eu/moredata" />
9	<meta http-equiv="content-type" content="text/html; charset=utf-8" />
10	<meta name="author" content="Paracrawl website manager" />
11	<meta name="description" content="ParaCrawl" />
12	<meta name="generator" content="Joomla! - Open Source Content Management" />
13	<title>More data</title>
14	<link href="/templates/shaper_helixultimate/images/favicon.ico" rel="shortcut icon" type="image/vnd.microsoft.icon" />
15	<link href="https://cdn.jsdelivr.net/npm/[email protected]/css/simple-line-icons.css" rel="stylesheet" type="text/css" />
16	<link href="/components/com_k2/css/k2.css?v=2.10.3&b=20200429" rel="stylesheet" type="text/css" />
17	<link href="//fonts.googleapis.com/css?family=Roboto:100,100i,300,300i,400,400i,500,500i,700,700i,900,900i&amp;display=swap&amp;subset=cyrillic-ext" rel="stylesheet" type="text/css" />
18	<link href="/templates/shaper_helixultimate/css/bootstrap.min.css" rel="stylesheet" type="text/css" />
19	<link href="/templates/shaper_helixultimate/css/font-awesome.min.css" rel="stylesheet" type="text/css" />
20	<link href="/templates/shaper_helixultimate/css/fa-v4-shims.css" rel="stylesheet" type="text/css" />
21	<link href="/templates/shaper_helixultimate/css/template.css" rel="stylesheet" type="text/css" />
22	<link href="/templates/shaper_helixultimate/css/presets/default.css" rel="stylesheet" type="text/css" />
23	<style type="text/css">
24.jumbotron {
25margin: 0px;
26border-radius: 0px;
27background-color: inherit;
28background: url(/images/bg.jpg) no-repeat right center;
29font-weight: bold;
30}
31
32.btn-primary {
33background-color: #00276a;
34font-weight: bold;
35padding: 10px 20px;
36}
37
38.btn-primary:hover {
39background-color: #000000;
40}
41
42.main-menu {
43font-size: 0.45em;
44font-weight: bold;
45background-color: #00276a;
46color: #FFFFFF;
47border-radius: 10px 10px 0px 0px;
48width: fit-content;
49padding: 0px;
50display: inline-block;
51vertical-align: bottom;
52margin-bottom: 1px;
53}
54
55.main-menu a {
56font-size: 1.8em;
57color: #FFFFFF !important;
58}
59
60.main-menu ul li{
61padding: 8px 16px;
62}
63
64.main-menu ul li:hover {
65background-color: #A0A0A0;
66}
67
68.main-menu ul li:first-child{
69border-radius: 10px 0px 0px 0px;
70}
71
72.main-menu ul li:last-child{
73border-radius: 0px 10px 0px 0px;
74}
75
76.main-menu ul li:not(:first-child){
77border-left: 2px solid #FFFFFF;
78}
79
80#sp-topheader {
81height: fit-content;
82}
83
84#sp-top1,
85#sp-top1 .sp-column,
86#sp-top1 .sp-module,
87#sp-top1 .sp-module-content {
88display: flex;
89width: 100%;
90}
91
92.logo-image {
93margin: 10px;
94height: 80px !important;
95}
96
97@media all and (max-width: 991.98px)
98{
99    .jumbotron
100    {
101        height: fill-content;
102        background-size: 100%;
103        padding: 1.2rem;
104    }
105   h1 {
106        font-size: 1.5rem !important;
107   }
108}
109
110#sp-right .sp-module {
111padding: 0px;
112}
113
114#sp-copyright {
115font-size: 14px;
116}
117
118.nav-item.active {
119background-color: #666666;
120}
121
122.sp-module {
123margin-top: auto !important;
124}
125
126.catItemView {
127border-bottom: 1px solid lightgrey;
128}�body{font-family: 'Roboto', sans-serif;font-size: 16px;font-style: normal;}
129@media (min-width:768px) and (max-width:991px){body{font-size: 14px;}
130}
131@media (max-width:767px){body{font-size: 14px;}
132}
133�.logo-image {height:36px;}.logo-image-phone {height:36px;}�#sp-section-2{ background-color:#0270bc;color:#ffffff; }�#sp-main-body{ background-image:url("/images/bg2.png");background-repeat:no-repeat;background-size:cover;background-attachment:fixed;background-color:#e0e0e0; }
134	</style>
135	
135<script src="/media/jui/js/jquery.min.js?b00ea91c4b4a04d386366e78a59f45db" type="text/javascript"></script>
135
136	
136<script src="/media/jui/js/jquery-noconflict.js?b00ea91c4b4a04d386366e78a59f45db" type="text/javascript"></script>
136
137	
137<script src="/media/jui/js/jquery-migrate.min.js?b00ea91c4b4a04d386366e78a59f45db" type="text/javascript"></script>
137
138	
138<script src="/media/k2/assets/js/k2.frontend.js?v=2.10.3&b=20200429&sitepath=/" type="text/javascript"></script>
138
139	
139<script src="/media/system/js/caption.js?b00ea91c4b4a04d386366e78a59f45db" type="text/javascript"></script>
139
140	
140<script src="/templates/shaper_helixultimate/js/popper.min.js" type="text/javascript"></script>
140
141	
141<script src="/templates/shaper_helixultimate/js/bootstrap.min.js" type="text/javascript"></script>
141
142	
142<script src="/templates/shaper_helixultimate/js/main.js" type="text/javascript"></script>
142
143	
143<script type="text/javascript">
144jQuery(window).on('load',  function() {
145				new JCaption('img.caption');
146			});�jQuery(document).ready(function (){
147jQuery("#k2Container").addClass("bg-white p-3");
148});�template="shaper_helixultimate";
149	</script>
vendor: 1 bytes, line 149
149
150<script src="https://kit.fontawesome.com/d3a98684a9.js" crossorigin="anonymous">
151</script>
151
152    </head>
153    <body class="site helix-ultimate com-content view-article layout-default task-none itemid-109 en-gb ltr sticky-header layout-fluid offcanvas-init offcanvs-position-right">
154    
155    <div class="body-wrapper">
156        <div class="body-innerwrapper">
157                        <section id="sp-topheader"><div class="container"><div class="container-inner"><div class="row"><div id="sp-top1" class="col-lg-4 col-xl-4 "><div class="sp-column "><div class="sp-module "><div class="sp-module-content"><div class="d-flex align-items-center justify-content-center justify-content-lg-start w-100">
158  <img src="/images/logo.png" class="logo-image" />
159</div></div></div></div></div><div id="sp-top2" class="col-12 col-lg-8 col-xl-8 "><div class="sp-column "><div class="sp-module "><div class="sp-module-content"><div class="d-flex justify-content-center justify-content-lg-end">
160<div class="d-inline-flex flex-column">
161    <div class="d-flex justify-content-center">
162        <a href="https://ec.europa.eu/inea/en/connecting-europe-facility" target="_blank">
163            <image src="/images/logo_en_cef.png" style="max-width: 420px; margin: 5px"/>
164        </a>
165    </div>
166    <div class="d-flex justify-content-center">
167    <nav class="navbar navbar-expand main-menu">
168        <ul class="navbar-nav">
169                    <li class="nav-item px-3 ">
170                    <a href="/">Releases</a>
171            </li>
172                    <li class="nav-item px-3 ">
173                    <a href="/about">About</a>
174            </li>
175                    <li class="nav-item px-3 active">
176                    <a href="/moredata">More data</a>
177            </li>
178                    <li class="nav-item px-3 ">
179                    <a href="/component/k2/itemlist/category/1">News</a>
180            </li>
181                </ul>
182    </nav>
183    </div>
184</div>
185</div></div></div></div></div></div></div></div></section><section id="sp-section-2"><div class="row"><div id="sp-title" class="col-lg-12 "><div class="sp-column "><div class="sp-module "><div class="sp-module-content"><section class="jumbotron text-center text-md-left p-3">
186    <div class="container">
187      <div class="col-lg-6 col-md-8 col-12">
188        <h1 class="jumbotron-heading" style="font-size: 1.8em">Broader/Continued Web-Scale Provision of Parallel Corpora for European Languages</h1>
189      </div>
190    </div>
191</section>
192</div></div><div class="sp-module "><div class="sp-module-content"><div style="display: flex; justify-content: center; padding: 10px; align-items: center" class="bg-white">
193    <a class="" href="https://www.ed.ac.uk" target="_blank" style="margin: 0px 10px;">
194        <img src="/images/logouoedin2.png" alt="" height="42">
195    </a>
196    <a class="" href="https://www.ua.es/en" target="_blank" style="margin: 0px 10px;">
197        <img src="/images/logoua2.jpg" alt="" height="38">
198    </a>
199    <a class="" href="https://www.taus.net" target="_blank" style="margin: 0px 10px;">
200        <img src="/images/logotaus.jpg" alt="" height="32">
201    </a>
202    <a class="" href="http://www.prompsit.com" target="_blank" style="margin: 0px 10px;">
203        <img src="/images/logoprompsit2.png" alt="" height="45">
204    </a>
205    <a class="" href="https://omniscien.com" target="_blank" style="margin: 0px 10px;">
206        <img src="/images/logoomniscien.png" alt="" height="45">
207    </a>
208</div></div></div></div></div></div></section><section id="sp-main-body"class="p-3"><div class="container"><div class="container-inner"><div class="row"><main id="sp-component" class="col-lg-12 " role="main"><div class="sp-column p-0 h-100"><div id="system-message-container">
209	</div>
210<div class="sp-module-content-top clearfix"><div class="sp-module "><div class="sp-module-content"><div class="card mb-3">
211<div class="card-body">
212<h4 class="card-title">More monolingual and parallel data</h4>
213<p class="card-text big">This section is about additional sources of parallel or monolingual data done using parts or the whole ParaCrawl pipeline or using the data to create derived corpora.
214</div>
215</div>
216
217
218
219
220<div class="card mb-3">
221<div class="row no-gutters">
222<div class="col-lg-3 text-center bg-light">
223<img src="/images/logo138x65.png" class="img-responsive m-2 m-lg-4 w-75 mx-auto"  style="mix-blend-mode: multiply;"/>
224
225<div class="m-3">
226<div class="progress">
227  <div class="progress-bar bg-info" role="progressbar" style="width: 100%;" aria-valuenow="100" aria-valuemin="0" aria-valuemax="100"><strong>100%</strong></div>
228</div>
229</div>
230
231
232<div class="m-3">
233<div class="small">Last release</div>
234<div class="badge badge-secondary"><strong>November 2024</strong></div><div class="small"> English-Asian languages</div>
235</div>
236</div>
237<div class="col-lg-9">
238<div class="card-body">
239<h5 class="card-title"> South and East Asian Languages bonus corpus out! </h5>
240
241<p class="card-text">The bonus corpus includes 9 language pairs: Hindi, Nepali, Burmese, Thai, Lao, Khmer, Vietnamese, Indonesian, and Korean paired with English. 
242Building on the work of the Paracrawl project, it follows the same general sequence of steps although different technologies are applied: targeted web crawling, document alignment (using efficient <a href="https://aclanthology.org/P18-4020/">Marian neural models</a> distilled from <a href="https://arxiv.org/abs/2207.04672">NLLB<a/>), sentence alignment (using <a href="https://aclanthology.org/D19-1136/">Vecalign</a>), and parallel corpus filtering (using <a href="https://aclanthology.org/W19-5435/">LASER</a> 3), including deduplication and a novel toxicity filter. 
243
244<h6 class="card-title">Downloading and formats</h6>
245<p class="card-text">The new parallel corpora can be downloaded from the bonus section in the <a href="https://paracrawl.eu/">Releases</a> tab and are provided in three different formats: txt (only parallel sentences, one per line), tmx (filtered parallel sentences and metadata in xml format) and dedup (filtered parallel sentences and metadata in a tab-separated file format).</p>
246
247More info at: https://www2.statmt.org/wmt24/pdf/2024.wmt-1.132.pdf 
248</div>
249</div>
250</div>
251</div> 
252
253
254<div class="m-3">
255<div class="small">Last release</div>
256<div class="badge badge-secondary"><strong>October 2022</strong></div><div class="small"> Polish-Czech</div>
257</div>
258</div>
259<div class="col-lg-9">
260<div class="card-body">
261<h5 class="card-title"> Polish-Czech bonus corpus released! </h5>
262
263<p class="card-text">The corpus has 24M bilingual segments.  Text was <a href="https://github.com/bitextor/warc2text">extracted from HTML</a>, <a href="https://github.com/CLD2Owners/cld2">classified by language</a>, <a href="https://github.com/kpu/preprocess/blob/master/moses/ems/support/split-sentences.perl">split</a>, <a href="https://github.com/kpu/preprocess/blob/master/preprocess/dedupe_main.cc">deduplicated</a> and  double-side filtered with <a href="https://github.com/bitextor/bicleaner-ai">Bicleaner-ai</a>.</p>
264
265<h6 class="card-title">Downloading and formats</h6>
266<p class="card-text">The Polish-Czech parallel corpus can be downloaded from the bonus section in the <a href="https://paracrawl.eu/">release</a> tab and is provided in three different formats: <a href="http://web-language-models.s3-website-us-east-1.amazonaws.com/paracrawl/bonus/pl-cs-v1.txt.gz">txt</a> (only parallel sentences, one per line), <a href="http://web-language-models.s3-website-us-east-1.amazonaws.com/paracrawl/bonus/pl-cs-v1.tmx.gz">tmx</a> (filtered parallel sentences and metadata in xml format) and <a href="http://web-language-models.s3-website-us-east-1.amazonaws.com/paracrawl/bonus/pl-cs-v1.classified.gz">raw</a> (unfiltered parallel sentences and metadata).</p>
267
268</div>
269</div>
270</div>
271</div> 
272
273
274
275
276<div class="card mb-3">
277<div class="row no-gutters">
278<div class="col-lg-3 text-center bg-light">
279<img src="/images/logo138x65.png" class="img-responsive m-2 m-lg-4 w-75 mx-auto"  style="mix-blend-mode: multiply;"/>
280
281<div class="m-3">
282<div class="progress">
283  <div class="progress-bar bg-info" role="progressbar" style="width: 100%;" aria-valuenow="100" aria-valuemin="0" aria-valuemax="100"><strong>100%</strong></div>
284</div>
285</div>
286
287<div class="m-3">
288<div class="small">Bonus release</div>
289<div class="badge badge-secondary"><strong>June 2022</strong></div><div class="small">Language Classified Web Text from ParaCrawl</div>
290</div>
291</div>
292<div class="col-lg-9">
293<div class="card-body">
294<h5 class="card-title">Language Classified Web Text from ParaCrawl</h5>
295
296
297<p class="card-text">This is a release of text from Internet Archive and targeted crawls performed in the ParaCrawl project. Text was extracted from web pages with document-level language classification provided by a <a href="https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.218a.bin">fasttext model</a>,  for <a href="https://github.com/bitextor/warc2text">189 languages</a>.</p>
298
299<p class="card-text"> There is a separate directory for each data source: wide00016, wide00015, wide00006, GWB-20191109192916, hieu, marta, and philipp.  Large crawls from the Internet Archive are wide00016, wide00015, wide00006. GWB-20191109192916 was extracted from a larger collection at the Internet Archive to match language codes ga, no, nb, nn, hr, and is in URLs with a corresponding URL containing en, but incidentally collected some other languages too.  Hieu and Philipp aimed at sites with a mix of languages and Marta aimed at languages in Spain.  The project also used <a href="https://commoncrawl.org/">CommonCrawl</a> which is already public.</p>
300
301<p class="card-text">The special language code <code>und</code> contains documents that were not at least 0.5 in any one language code according to the classifier. This is mainly provided so that the set of documents is complete and one could run their own language identifier over the entire corpus. Currently wide00015 does not have an <code>und</code> directory pending supercomputer time; the rest are complete.</p>
302
303<p class="card-text">A complete list of files is in <a href="https://web-language-models.s3.us-east-1.amazonaws.com/paracrawl/monolingualv9/files">files</a>. To download all files, run:</p>
304
305<pre>
306wget https://web-language-models.s3.us-east-1.amazonaws.com/paracrawl/monolingualv9/files
307wget -nH -x --cut-dirs=2 -i files
308</pre>
309
310<p class="card-text">Within each collection, there are separate directories by ISO 639-3 language code and then a text.gz and a url.gz. Each line in text.gz is a base64-encoded document while the corresponding line in url.gz is the URL. Once decoded, the document is in UTF-8 with just text extracted from a web page with newlines added at block element boundaries; we recommend running your own sentence splitter configured to insert but not remove newlines.</p>
311
312<p class="card-text">Example to just print text: <code>zcat text.gz |base64 -d</code>. A longer example of python:</p>
313
314
315<pre>
316#!/usr/bin/env python3
317import gzip
318import base64
319
320with gzip.open("text.gz") as texts, gzip.open("url.gz") as urls:
321  for encoded, url in zip(texts, urls):
322    text = base64.standard_b64decode(encoded).decode('utf-8')
323    url = url.decode('utf-8')
324    print("Contents of " + url)
325    print(text)
326
327</pre>
328
329
330<p class="card-text">However, for performance we recommend using <a href="https://zlib.net/pigz/">pigz</a> for decompression and <a href="https://www.gnu.org/software/parallel/">GNU parallel</a> to parallelize over documents</p>
331<h6 class="card-title">Next processing step advice</h6>
332<p class="card-text">Run a sentence splitter then you'll probably want to classify language at sentence level. English UI elements tend to creep into every language.</p>
333
334
335</div>
336</div>
337</div>
338</div> 
339
340
341
342<div class="card mb-3">
343<div class="row no-gutters">
344<div class="col-lg-3 text-center bg-light">
345<img src="/images/pc3.png" class="img-responsive m-2 m-lg-4 w-75 mx-auto"  style="mix-blend-mode: multiply;"/>
346
347<div class="m-3">
348<div class="progress">
349  <div class="progress-bar bg-info" role="progressbar" style="width: 100%;" aria-valuenow="100" aria-valuemin="0" aria-valuemax="100"><strong>100%</strong></div>
350</div>
351</div>
352
353
354<div class="m-3">
355<div class="small">Last release</div>
356<div class="badge badge-secondary"><strong>April 2021</strong></div><div class="small">English monolingual data from ParaCrawl V8</div>
357</div>
358</div>
359<div class="col-lg-9">
360<div class="card-body">
361<h5 class="card-title">Monolingual data from ParaCrawl V8: English</h5>
362
363<p class="card-text">The corpus has 96,470,655,818 lines, 1,337,127,886,176 tokens, and 9,153,226,323,307 characters of English.  Text was <a href="https://github.com/bitextor/warc2text">extracted from HTML</a>, <a href="https://github.com/CLD2Owners/cld2">classified</a>, <a href="https://github.com/kpu/preprocess/blob/master/moses/ems/support/split-sentences.perl">split</a>, and <a href="https://github.com/kpu/preprocess/blob/master/preprocess/dedupe_main.cc">deduplicated</a>.</p>
364
365<p class="card-text">The corpus is available as 128 files, <a href="https://github.com/kpu/preprocess/blob/master/preprocess/shard_main.cc">split by the hash of the line</a>.  The first and last URLs are: <br/>
366<a href="https://neural.mt/data/paracrawl8-mono/en-000.gz">https://neural.mt/data/paracrawl8-mono/en-000.gz</a><br/>
367<a href="https://neural.mt/data/paracrawl8-mono/en-127.gz">https://neural.mt/data/paracrawl8-mono/en-127.gz</a></p>
368<pre>
369#!/bin/bash
370for i in {0..127}; do
371  wget https://neural.mt/data/paracrawl8-mono/en-$(printf "%03i" $i).gz
372done
373</pre>
374<p class="card-text">Files are hosted on the <a href="https://archive.org/">Internet Archive</a>.  Due to their 1 TB limit per directory, there are redirects to the appropriate directory.</p>
375
376<h6 class="card-title">Source data</h6>
377
378This is all the English data used for ParaCrawl release 8, which is based on the following crawls.
379<ul>
380<li><strong>Internet Archive</strong>: <span>
380<a href="https://archive.org/details/wide00006">wide00006</a>, <a href="https://archive.org/details/wide00015">wide00015</a>, and <a href="https://archive.org/details/GWB-20191109192916_multi-lang_en-is-hr-no-ga_WARCs">pages with en, is, hr, no, and ga in their URL</a>.</span></li>
381
382<li><strong>CommonCrawl</strong>: <span><a href="https://commoncrawl.org/2016/08/july-2016-crawl-archive-now-available/">2016-30</a>, <a href="https://commoncrawl.org/2017/07/july-2017-crawl-archive-now-available/">2017-30</a>, <a href="https://commoncrawl.org/2018/07/3-25-billion-pages-crawled-in-july-2018/">2018-30</a>, <a href="https://commoncrawl.org/2019/04/april-2019-crawl-archive-now-available/">2019-18</a>, and <a href="https://commoncrawl.org/2019/08/august-2019-crawl-archive-now-available/">2019-35</a>.<span></li>
383
384<li><strong>Targeted</strong>: <span><a href="http://www.cs.jhu.edu/~phi/">Philipp Koehn</a> crawled domains that have a mix of multilingual content based on language classification in CommonCrawl.  <a href="https://about.me/motagirl2">Marta Bañón</a> aimed for sites in Basque, Catalan, Galician, and Spanish but picked up some English on the way.  <a href="http://statmt.org/hieu/">Hieu Hoang</a> crawled sites that produced parallel sentences in earlier generations of ParaCrawl.<span></li>
385</ul>
386
387<h6 class="card-title">More languages</h6>
388<p class="card-text">Coming, though ParaCrawl release 9 processing takes priority.  That will have even more data!</p>
389
390</div>
391</div>
392</div>
393</div> 
394
395
396<div class="card mb-3">
397<div class="row no-gutters">
398<div class="col-lg-3 text-center bg-light">
399<img src="/images/EuroPatLogo_blue.png" class="img-responsive m-2 m-lg-4 w-75 mx-auto"  style="mix-blend-mode: multiply;"/>
400
401<div class="m-3">
402<div class="progress">
403  <div class="progress-bar bg-info" role="progressbar" style="width: 100%;" aria-valuenow="100" aria-valuemin="0" aria-valuemax="100"><strong>100%</strong></div>
404</div>
405</div>
406<div class="m-3">
407<div class="small">Next Data Releases</div>
408<div class="badge badge-secondary"><strong>September 2021</strong></div>
409<div class="small">Patent parallel corpora made of English and Croatian, Norwegian (Bokmål), German, Polish, Spanish, and French. Also Icelandic might be included</div>
410</div>
411</div>
412<div class="col-lg-9">
413<div class="card-body">
414<h5 class="card-title">EuroPat: Unleashing European Patent Translations</h5>
415<p class="card-text">Patents provide a rich source of technical vocabulary, product names, and person names that complement other data sources used for machine translation.</p><p>This Action will will mine parallel corpora from patents by aggregating, aligning, and converting patent data. Alignment and cleaning modules in the ParaCrawl pipeline will be enhanced and used to carry out this action.</p><p>The first release included English-German (12.6M parallel sentences) and English-French corpora (9.2M parallel sentences) made up by using information from the European Patent Organisation database to identify patents.</p><p>The second release includes 6 language combinations: English-German (15.5M parallel sentences), English-Spanish (44.4M parallel sentences), English-French corpora (12M parallel sentences), English-Croatian (75k parallel sentences), English-Norwegian (4M parallel sentences) and English-Polish (89k parallel sentences) from various sources.</p>
416<p class="card-text">
417<small class="text-muted">Implementation schedule: September 2010 to September 2021</small></p>
418<a href="https://ec.europa.eu/inea/en/connecting-europe-facility/cef-telecom/2018-eu-ia-0061" target="_blank" class="btn btn-sm btn-secondary"><strong><i class="fas fa-external-link-alt fa-sm">&nbsp;</i> More info</strong></a>
419
420<a href="https://europat.net/" target="_blank" class="btn btn-sm btn-secondary"><strong><i class="fas fa-external-link-alt fa-sm">&nbsp;</i> Project website</strong></a>
421
422
423<a href="https://archive.org/download/Europat-v2.0" target="_blank" class="btn btn-sm btn-secondary"><strong><i class="fas fa-external-link-alt fa-sm">&nbsp;</i>Download the data</strong></a>
424
425</div>
426</div>
427</div>
428</div>
429
430
431<div class="card mb-3">
432<div class="row no-gutters">
433<div class="col-lg-3 text-center bg-light">
434<img src="/images/opus_medium.png" class="img-responsive m-2 m-lg-4 w-75 mx-auto"  style="mix-blend-mode: multiply;"/>
435
436<div class="m-3">
437<div class="progress">
438  <div class="progress-bar bg-success" role="progressbar" style="width: 100%;" aria-valuenow="100" aria-valuemin="0" aria-valuemax="100"><strong>Completed</strong></div>
439</div>
440</div>
441
442
443<div class="m-3">
444<div class="small">Last release</div>
445<div class="badge badge-secondary"><strong>October 2020</strong></div><div class="small">Multi-parallel corpus by pivoting via English made from ParaCrawl data.</div>
446</div>
447</div>
448<div class="col-lg-9">
449<div class="card-body">
450<h5 class="card-title">MultiParaCrawl v 7.1</h5>
451<p class="card-text">Parallel corpora from Web Crawls collected in the ParaCrawl project and further processed for making it a multi-parallel corpus by pivoting via English. They only provide the additional language pairs that came out of pivoting. The bitexts for English are available from the ParaCrawl release.</br>
452
453Stats about the data in MultiParaCrawl v7.1: 
454<ul>
455<li>40 languages, 669 bitexts</li>
456<li>total number of files: 40</li>
457<li>total number of tokens: 10.14G</li>
458<li>total number of sentence fragments: 505.48M</li>
459</ul>
460</p>
461
462<a href="http://opus.nlpl.eu/index.php" target="_blank" class="btn btn-sm btn-secondary"><strong><i class="fas fa-external-link-alt fa-sm">&nbsp;</i> OPUS website</strong></a>
463<a href="http://opus.nlpl.eu/MultiParaCrawl.php" target="_blank" class="btn btn-sm btn-secondary"><strong><i class="fas fa-external-link-alt fa-sm">&nbsp;</i>Download the data</strong></a>
464</div>
465</div>
466</div>
467</div>
468</div></div></div><div class="article-details " itemscope itemtype="https://schema.org/Article">
469	<meta itemprop="inLanguage" content="en-GB">
470	
471					
472	
473		
474		<div class="article-can-edit d-flex flex-wrap justify-content-between">
475				</div>
476	
477		
478			
479		
480		
481			<div class="article-ratings-social-share d-flex justify-content-end">
482			<div class="mr-auto align-self-center">
483							</div>
484			<div>
485				<div class="article-social-share">
486	<div class="social-share-icon">
487		<ul>
488											<li>
489					<a class="facebook" onClick="window.open('https://www.facebook.com/sharer.php?u=https://paracrawl.eu/moredata','Facebook','width=600,height=300,left='+(screen.availWidth/2-300)+',top='+(screen.availHeight/2-150)+''); return false;" href="https://www.facebook.com/sharer.php?u=https://paracrawl.eu/moredata" title="Facebook">
490						<span class="fa fa-facebook"></span>
491					</a>
492				</li>
493																											<li>
494					<a class="twitter" title="Twitter" onClick="window.open('https://twitter.com/share?url=https://paracrawl.eu/moredata&amp;text=More%20parallel%20data','Twitter share','width=600,height=300,left='+(screen.availWidth/2-300)+',top='+(screen.availHeight/2-150)+''); return false;" href="https://twitter.com/share?url=https://paracrawl.eu/moredata&amp;text=More%20parallel%20data">
495						<span class="fa fa-twitter"></span>
496					</a>
497				</li>
498																												<li>
499						<a class="linkedin" title="LinkedIn" onClick="window.open('https://www.linkedin.com/shareArticle?mini=true&url=https://paracrawl.eu/moredata','Linkedin','width=585,height=666,left='+(screen.availWidth/2-292)+',top='+(screen.availHeight/2-333)+''); return false;" href="https://www.linkedin.com/shareArticle?mini=true&url=https://paracrawl.eu/moredata" >
500							<span class="fa fa-linkedin-square"></span>
501						</a>
502					</li>
503										</ul>
504		</div>
505	</div>
506			</div>
507		</div>
508	
509	<div itemprop="articleBody">
510			</div>
511
512	
513	
514				
515				
516		
517	
518
519	
520				</div>
521
522</div></main></div></div></div></section><footer id="sp-footer"class="bg-white text-dark"><div class="container"><div class="container-inner"><div class="row"><div id="sp-footer1" class="col-lg-6 "><div class="sp-column p-3 pb-0 "><div class="sp-module "><div class="sp-module-content"><div>
523<strong>Action 3: Continued Web-Scale Provision of Parallel Corpora for European Languages</strong>
524<ul>
525<li>September 2020, Augmented Data Release including Manufactured Data</li>
526<li>September 2020, Implementation of Deferred Crawling</li>
527<li>March 2021, Data Release 1 (This will include extra data from the internet archive and an updated version of the manufactured data)</li>
528<li>May 2021, Completion of data portal, with data from release 1</li>
529<li>September 2021, Data Release 2 (This will include more data from the internet archive and an updated version of the manufactured data)</li>
530<li>September 2021, Code Release</li>
531<li>September 2021, Quality testing for release 1</li>
532<li>September 2021, Data promotion plan</li>
533<li>September 2021, Validation: Metadata related to "Augmented Data, Data release 1 and 2" to be compliant with the ELRC SHARE specifications and available on the ELRC-SHARE repository</li>
534</ul>
535</div></div></div><div class="sp-module "><div class="sp-module-content"><div>
536<strong>Action 2: Broader Web-Scale Provision of Parallel Corpora for European Languages</strong>
537<ul>
538<li>Novermber 2018, Website updated with Broader WebCrawl</li>
539<li>February 2019, Integration of Initial Domain Identification Technology</li>
540<li>June 2019, Integration of Processing of Broader Document Formats</li>
541<li>September 2019, Inclusion of Data from Internet Archive in Data Release</li>
542<li>September 2019, Data Release 1</li>
543<li>March 2020, Data Release 2</li>
544<li>August 2020, Final Code Release</li>
545<li>August 2020, Domain Identification for Data Release 1</li>
546<li>September 2020, Data Release 3</li>
547</ul>
548</div></div></div><div class="sp-module "><div class="sp-module-content"><div>
549<strong>Action 1: Provision of Web-Scale Parallel Corpora for Official European Languages</strong>
550<ul>
551<li>September 2017, Kickoff meeting in Alicante, Spain</li>
552<li>November 2017, Website up and running</li>
553<li>January 2018, Release of Corpus v1</li>
554<li>April 2018, Release of Corpus v1 in ELRC-SHARE</li>
555<li>June 2018, Release of Corpus v2</li>
556<li>
556June 2018, Release of Software v1</li>
557<li>October 2018, Release of Corpus v3</li>
558<li>March 2019, Release of Corpus v4, also in ELRC-SHARE</li>
559<li>March 2019, Release of Software v2</li>
560</ul>
561</div></div></div></div></div><div id="sp-footer2" class="col-lg-6 "><div class="sp-column  p-3 pb-0"><div class="sp-module "><div class="sp-module-content"><div class="d-flex justify-content-center justify-content-lg-start">
562   <div class="d-flex align-items-center">
563      <a href="https://www.paracrawl.eu">
564      <img class="m-2" src="/images/logo135x62.png" alt="" width="134">
565      </a>
566   </div>
567   <div class="d-flex align-items-center">
568      <a href="https://ec.europa.eu/inea/en/connecting-europe-facility" target="_blank">
569      <img class="m-2" src="/images/logo_en_cef273x39.png" alt="" width="273">
570      </a>
571   </div>
572</div>
573<hr>
574<div class="text-center text-lg-left mt-4 mb-2"><strong>Project Partners</strong></div>
575<div class="d-flex justify-content-center justify-content-lg-start">
576   <div class="d-flex align-items-center">
577      <a href="https://www.ed.ac.uk" target="_blank">
578      <img class="m-2"  src="/images/logouoedin.png?" alt="" width="159">
579      </a>
580   </div>
581   <div class="d-flex align-items-center">
582      <a href="https://www.ua.es/en" target="_blank">
583      <img class="m-2" src="/images/logoua.jpg" alt="" width="153">
584      </a>
585   </div>
586</div>
587<div class="d-flex justify-content-center justify-content-lg-start">
588   <div class="d-flex align-items-center">
589      <a href="https://www.taus.net" target="_blank">
590      <img class="m-2" src="/images/logotaus.png" alt="" width="120">
591      </a>
592   </div>
593   <div class="d-flex align-items-center">
594      <a href="http://www.prompsit.com/" target="_blank">
595      <img class="m-2" src="/images/logoprompsit.png" alt="" width="110">
596      </a>
597   </div>
598   <div class="d-flex align-items-center">
599      <a href="https://omniscien.com" target="_blank">
600      <img class="m-2" src="/images/logoomniscien.png" alt="" width="130">
601      </a>
602   </div>
603</div>
604<hr>
605<div class="text-center text-lg-left mt-4 mb-2"><strong>Other Contributors</strong></div>
606<div class="d-flex justify-content-center justify-content-lg-start">
607   <div class="d-flex align-items-center">
608      <a href="https://www.jhu.edu/" target="_blank">
609      <img class="m-2" class="" src="/images/logojh.png" alt="" width="120">
610      </a>
611   </div>
612   <div class="d-flex align-items-center">
613      <a href="http://www.mozilla.org" target="_blank">
614      <img class="m-2" class="" src="/images/logomozilla.jpg" alt="" width="84">
615      </a>
616   </div>
617   <div class="d-flex align-items-center">
618      <a href="https://registry.opendata.aws/paracrawl/" target="_blank">
619      <img class="m-2" class="" src="/images/aws-logo.png" alt="" width="60">
620      </a>
621   </div>
622</div></div></div><div class="sp-module "><div class="sp-module-content"><hr>
623<div class="text-left">
624<strong >Acknowledging CSD3</strong>
625<p>
626ParaCrawl Releases 6 and later were processed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (<a href="http://www.csd3.cam.ac.uk/" target="_blank" class="alert-link text-secondary">www.csd3.cam.ac.uk</a>), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/P020259/1), and DiRAC funding from the Science and Technology Facilities Council (<a href="http://www.dirac.ac.uk/" class="alert-link text-secondary">www.dirac.ac.uk</a>).
627</p>
628</div></div></div><div class="sp-module "><div class="sp-module-content"><hr>
629<div class="text-left">
630<strong >Acknowledging Internet Archive</strong>
631<p>
632ParaCrawl includes data from the Internet Archive (<a href="https://archive.org/" target="_blank" class="alert-link text-secondary">https://archive.org/</a>) as part of an agreement between the Internet Archive and the University of Edinburgh.
633</p>
634</div></div></div><div class="sp-module "><div class="sp-module-content"><hr>
635<div class="text-left">
636<strong >Acknowledging ODNI and IARPA</strong>
637<p>
638This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract # FA8650-17-C-9117. The views and conclusions contained herein are those of the authors and should not
639be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S.
640Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
641</p>
642</div></div></div></div></div></div></div></div></footer><section id="sp-copyright"class="p-4 pt-0"><div class="container"><div class="container-inner"><div class="row"><div id="sp-position8" class="col-lg-12 "><div class="sp-column "><div class="sp-module "><div class="sp-module-content"><hr>
643<div class="text-center text-lg-left">
644<div>Broader/Continued Web-Scale Provision of Parallel Corpora for European Languages</div>
645<div>© Copyright 2019. All rights reserved.</div>
646<div style="line-height: normal;"><small>Any communication or publication related to the action, made by the beneficiaries jointly or individually in any form and using any means, shall indicate that it reflects only the author's view and that the Agency is not responsible for any use that may be made of the information it contains.</small></div>
647</div></div></div></div></div></div></div></div></section>        </div>
648    </div>
649
650    <!-- Off Canvas Menu -->
651    <div class="offcanvas-overlay"></div>
652    <div class="offcanvas-menu">
653        <a href="#" class="close-offcanvas"><span class="fa fa-remove"></span></a>
654        <div class="offcanvas-inner">
655                            <p class="alert alert-warning">
656                    Publish the Menu module to &quot;offcanvas&quot; position. Here you can publish other modules as well. <br /> <a target='_blank' href='https://www.joomshaper.com/documentation/helix-framework/helixultimate#documentation-heading-5_13'>Learn More.</a>                </p>
657                    </div>
658    </div>
659
660    
661    
662    
663    <!-- Go to top -->
664            <a href="#" class="sp-scroll-up" aria-label="Scroll Up"><span class="fa fa-chevron-up" aria-hidden="true"></span></a>
665            </body>
666</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.