-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathFineOPUS.html
More file actions
272 lines (246 loc) · 14.4 KB
/
Copy pathFineOPUS.html
File metadata and controls
272 lines (246 loc) · 14.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="description" content="FineOPUS: Refining the World’s Largest Parallel Corpus for the Future of AI">
<meta property="og:title" content="FineOPUS" />
<meta property="og:description" content="A Massively Multilingual Translation Corpus and Pipeline" />
<meta property="og:url" content="opus.nlpl.eu" />
<meta name="twitter:title" content="FineOPUS">
<meta name="twitter:description" content="Refining the World’s Largest Parallel Corpus for the Future of AI">
<meta name="twitter:card" content="summary_large_image">
<meta name="keywords" content="NLP, Machine Translation, Data Curation, LLM, Multilingual AI, OPUS">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>FineOPUS</title>
<link rel="icon" type="image/x-icon" href="static/images/favicon.ico">
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">
<link rel="stylesheet" href="static/css/bulma.min.css">
<link rel="stylesheet" href="static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="static/css/bulma-slider.min.css">
<link rel="stylesheet" href="static/css/fontawesome.all.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="static/css/index.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="static/js/fontawesome.all.min.js"></script>
<script src="static/js/index.js"></script>
</head>
<body>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">FineOPUS: A Massively Multilingual Translation Corpus
and Pipeline</h1>
<h3 class="subtitle is-4">Refining Parallel Texts in Many Languages</h3>
<div class="is-size-5 publication-authors">
<span class="author-block">
<a href="https://mala-lm.github.io" target="_blank">MaLA-LM Team</a>,
</span>
<span class="author-block">
<a href="https://www.ellisinstitute.fi" target="_blank">ELLIS Institute Finland</a>,
</span>
<span class="author-block">
<a href="https://turkunlp.org/" target="_blank">TurkuNLP</a>,
</span>
<span class="author-block">
<a href="https://blogs.helsinki.fi/language-technology/"
target="_blank">Helsinki-NLP</a>
</span>
</div>
<p>Nov 21, 2025</p>
<div class="column has-text-centered">
<div class="publication-links">
<span class="link-block">
<a href="https://opus.nlpl.eu/" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-globe"></i>
</span>
<span>Original OPUS</span>
</a>
</span>
<span class="link-block">
<a href="https://github.com/MaLA-LM/FineOPUS" class="external-link button is-normal is-rounded is-light">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span>
<span class="link-block">
<a href="https://huggingface.co/collections/MaLA-LM/fineopus" class="external-link button is-normal is-rounded is-light">
<span class="icon">
<span role="img" aria-label="Hugging Face Emoji">🤗</span>
</span>
<span>Dataset</span>
</a>
</span>
<!-- Discord link -->
<span class="link-block">
<a href="https://discord.com/invite/F5mEb7U6we" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-discord"></i>
</span>
<span>Discord</span>
</a>
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">The Mission: From Noise to Signal</h2>
<div class="content has-text-justified">
<p>
Data is the fuel of modern AI, but in the realm of multilingual models, <strong>quality is
quickly becoming more valuable than quantity</strong>.
</p>
<p>
For years, the <a href="https://opus.nlpl.eu/">OPUS collection</a> has served as the
backbone of open-source machine translation, offering a massive repository of parallel
texts. However, as we push the boundaries of what Large Language Models (LLMs) and Machine
Translation (MT) systems can do, the noise inherent in web-scraped and aggregated data has
become a bottleneck.
</p>
<p>
Today, we announce <strong>FineOPUS</strong>: a project dedicated to
transforming the vast OPUS collection into a foundational, high-quality parallel
corpus for the global AI community. Our mission is simple yet ambitious: we are applying a
rigorous, empirically-driven data curation philosophy to parallel data. Inspired by the
success of the FineWeb project, we aren't just cleaning data; we are systematically
engineering a resource to be reliable, equitable, and state-of-the-art.
</p>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">Why FineOPUS?</h2>
<div class="content has-text-justified">
<p>
Current open datasets often suffer from critical issues that hinder model performance:
</p>
<ul>
<li><strong>Semantic Misalignment:</strong> Sentences that don't actually mean the same
thing.</li>
<li><strong>Language Contamination:</strong> The wrong language appearing in a dataset
labeled for another.</li>
<li><strong>Formatting Artifacts:</strong> HTML tags and broken encoding that confuse
models.</li>
<li><strong>Inequity:</strong> A massive gap in quality and volume between high-resource
(e.g., English-French) and low-resource languages.</li>
</ul>
<p>
FineOPUS aims to mitigate these issues through a <strong>transparent, reproducible
pipeline</strong>.
</p>
</div>
</div>
</div>
</div>
</section>
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">The Methodology: Radical Empiricism</h2>
<div class="content has-text-justified">
<p>
We are not guessing which cleaning methods work best. We are proving it. The core of the
FineOPUS strategy is <strong>validation through ablation</strong>. We will train dozens of models to empirically justify every single design choice in our
pipeline. If a filtering step doesn't improve model performance, it doesn't make it into the
final pipeline.
</p>
<h4 class="title is-4">Our Multi-Stage Approach</h4>
<ul>
<li><strong>Scalable Cleaning:</strong> A pipeline encompassing normalization, language
re-identification, and neural parallelism filtering.</li>
<li><strong>Principled Quality Estimation:</strong> Moving beyond simple heuristics to
model-based quality controls.</li>
<li><strong>Targeted Augmentation:</strong> For low-resource languages, we aren't just
filtering (which reduces data); we are building. We will employ iterative
back-translation and synthetic data generation to bridge the gap for under-represented
languages.</li>
</ul>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">What We Are Delivering</h2>
<div class="content has-text-justified">
<p>
We believe in open science. Upon completion, FineOPUS will release three key assets to the
community:
</p>
<ol>
<li><strong>The FineOPUS Dataset:</strong> A large-scale, open-licensed parallel corpus
ready for training next-generation AI models.</li>
<li><strong>The FineOPUS Pipeline:</strong> The fully documented, reproducible open-source
code used to create the dataset.</li>
<li><strong>The Technical Report:</strong> A comprehensive account of our ablation studies,
providing transparency on <em>why</em> specific curation decisions were made.</li>
</ol>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Join Us on the Journey</h2>
<div class="content has-text-justified">
<p>
We are building FineOPUS to empower researchers and developers to build more powerful,
inclusive language technologies. By ensuring the world's linguistic diversity is represented
in the future of AI, we hope to bridge the digital divide one sentence pair at a time.
</p>
<p>
<strong>Stay tuned for our FineOPUS release and technical report.</strong>
</p>
<p>Discord: <a href="https://discord.com/invite/F5mEb7U6we">MaLA-LM</a></p>
</div>
</div>
</div>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This page was built using the <a
href="https://github.com/eliahuhorwitz/Academic-project-page-template"
target="_blank">Academic Project Page Template</a> which was adopted from the <a
href="https://nerfies.github.io" target="_blank">Nerfies</a> project page.
You are free to borrow the source code of this website, we just ask that you link back to
this page in the footer. <br> This website is licensed under a <a rel="license"
href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>
</div>
</div>
</div>
</div>
</footer>
</body>
</html>