Provision of Web-Scale Parallel Corpora for Official European Languages (ParaCrawl) Dataset for Machine Learning
Install DagsHub:
pip install dagshubTo stream this data directly on DagsHub
from dagshub.streaming import DagsHubFilesystem
fs = DagsHubFilesystem(".", repo_url="https://dagshub.com/DagsHub-Datasets/paracrawl-dataset")
fs.listdir("s3://web-language-models")Description
ParaCrawl is a set of large parallel corpora to/from English for all official EU languages by a broad web crawling effort. State-of-the-art methods are applied for the entire processing chain from identifying web sites with translated text all the way to collecting, cleaning and delivering parallel corpora that are ready as training data for CEF.AT and translation memories for DG Translation.
Additional information
Documentation
Update frequency
New data is added according to ParaCrawl release schedule.
Managed by
License
Creative Commons CC0 license (“no rights reserved”).
