http://comhis.fi/clusters/info/about

1<!doctype html>
2<html>
3
4<head>
5    <title>About</title>
6    

6<script src="https://code.jquery.com/jquery-3.1.1.min.js"></script>

6

7<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>

7
8<link href="https://bootswatch.com/3/spacelab/bootstrap.min.css" rel="stylesheet">
9</head>
10
11<body>
12    <nav class="navbar navbar-inverse">
13        <div class="container-fluid">
14            <div class="navbar-header">
15                <a class="navbar-brand" style="padding-left:50px;padding-right:24px" href="http://comhis.fi/clusters">Text Reuse in Finnish Newspapers and Journals, 1771–​1920</a>
16            </div>
17            <ul class="nav navbar-nav">
18                <li class="active"><a href="#">About</a></li>
19                <li><a href="/clusters/info/credits">Credits</a></li>
20                <li><a href="/clusters/info/cite">How to Cite</a></li>
21            </ul>
22        </div>
23    </nav>
24
25    <div class="container">
26			<div style="text-align:right">
27				<img src="http://comhis.fi/clusters/assets/blacklight/digihum-c4c4a3322f73a1e373e9831e7b0762d1ebb02ba8c68b644730e02eda6553c382.png" alt="" width=10% height=10%/>
28			</div>
29				<p>
30            <b>
31Text Reuse in Finnish Newspapers and Journals, 1771–1920</b>
32            <br />
33
34            <br /> The database Text Reuse in Finnish Newspapers and Journals, 1771–1920 has been created by the consortium Computational History and the Transformation of Public Discourse in Finland, 1640–1910 (funded by the Academy of Finland programme on Digital Humanities, 2016–2019). ​The consortium is based on the cooperation of four partners: The Faculty of Humanities at the University of Helsinki, the Departments of Cultural History and Future Technologies at the University of Turku and the Centre for Preservation and Digitisation of the National Library of Finland. This brings together relevant complementary expertise on the research subject (eighteenth- and nineteenth-century history), methodology (computational sciences and language technology) and data (the preservation and enhancement of digital resources). The objective of the consortium is to reassess the scope, nature, development, and transnational connections of public discourse in Finland, 1640–1910.
35            <br />
36            <br /> The database Text Reuse in Finnish Newspapers and Journals, 1771–1920 has been built by the researchers of the Department of Future Technologies and the Department of Cultural History, University of Turku. It is an outcome of the WP2 of the consortium, “Viral Texts and Social Networks of Finnish Public Discourse in Newspapers and Journals 1771–1910”. The main idea has been to identify repeated texts, or passages of texts, from a corpus of scanned and OCR-recognized Finnish newspapers and journals. Due to various reasons, like the 'Fraktur' font, the quality is at times fairly illegible. For this purpose, software was created for retrieving and exploring duplicated text passages in low quality OCR historical text corpora. The system uses NCBI BLAST, which is software created for comparing and aligning biological sequences, in order to find the duplicated passages. The web interface uses Solr as the back-end to perform the search queries and a modified version of Blacklight as the front-end.
37            <br />
38            <br /> The original OCR data is not perfectly segmented into articles, so elements such as page breaks, pictures etc. in the original image can cause multiple clusters of similar passages. Therefore, the absolute number of found passages does not necessarily reflect the actual level of reuse. The passages also have a minimum length requirement of 300 characters, as anything shorter often results in boilerplate.
39 	    <br />
40	    <br />
41	    For each cluster, we have also calculated a virality score. This score takes into account three factors: number of unique locations, number of unique titles and the time it took for the passage to circulate. We also do not consider outliers in the calculation, so that one or a few stray repetitions can not skew the results. Therefore by looking at the dates, we discard any repetitions that apper outside the Tukey fences.
42	    <br />
43	    The formula used: <i>locations*titles*(1/timespan) </i>, normalized to 0-100 range.
44            <br /> 
45            <br />
46            <br /> Further readings on the method and preliminary results:
47            <br />
48            <br /> Vesanto, Aleksi, Asko Nivala, Heli Rantala, Tapio Salakoski, Hannu Salmi & Filip Ginter, ’Applying BLAST to Text Reuse Detection in Finnish Newspapers and Journals, 1771–1910’, Proceedings of the 21st Nordic Conference of Computational Linguistics. Gothenburg, Sweden, 23–24 May 2017 (Linköping 2017), 54–58, <a href="http://www.ep.liu.se/ecp/133/010/ecp17133010.pdf">http://www.ep.liu.se/ecp/133/010/ecp17133010.pdf</a>
49            <br />
50            <br /> Vesanto, Aleksi, Asko Nivala, Tapio Salakoski, Hannu Salmi & Filip Ginter, ’A System for Identifying and Exploring Text Repetition in Large Historical Document Corpora’, Proceedings of the 21st Nordic Conference of Computational Linguistics. Gothenburg, Sweden, 23–24 May 2017 (Linköping 2017), 330–333, <a href="http://www.ep.liu.se/ecp/131/049/ecp17131049.pdf">http://www.ep.liu.se/ecp/131/049/ecp17131049.pdf</a>
51            <br />
52
53            <br />
54            <br /> Further information of the COMHIS consortium:
55            <br />
56            <a href="https://www.utu.fi/fi/yksikot/hum/yksikot/kulttuurihistoria/tutkimus/Sivut/comhis.aspx">https://www.utu.fi/fi/yksikot/hum/yksikot/kulttuurihistoria/tutkimus/Sivut/comhis.aspx</a>
57
58        </p>
59
60    </div>
61</body>
62
63</html>