Common Crawl

Stream data with DDA:

from dagshub.streaming import DagsHubFilesystem

fs = DagsHubFilesystem(".", repo_url="https://dagshub.com/DagsHub-Datasets/commoncrawl-dataset")

fs.listdir("s3://commoncrawl")

Description:

A corpus of web crawl data composed of over 50 billion web pages.

Contact:

A corpus of web crawl data composed of over 50 billion web pages.

Update Frequency:

Monthly

Managed By:

https://commoncrawl.org/

Resources:

1. resource:
   • Description: Crawl data (WARC and ARC format)
   • ARN: arn:aws:s3:::commoncrawl
   • Region: us-east-1
   • Type: S3 Bucket
   • AccountRequired: True

Tags:

aws-pds, encyclopedic, natural language processing, internet

Tutorials:

1. tutorial:
   • Title: Analysing Petabytes of Websites
   • URL: http://tech.marksblogg.com/petabytes-of-website-data-spark-emr.html
   • AuthorName: Mark Litwintschik
   • Services: EMR
2. tutorial:
   • Title: Index to WARC Files and URLs in Columnar Format
   • URL: https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/
   • AuthorName: Sebastian Nagel
   • Services: Athena
3. tutorial:
   • Title: Common Crawl Index Athena
   • URL: https://skeptric.com/common-crawl-index-athena/
   • AuthorName: Edward Ross
   • Services: Athena
4. tutorial:
   • Title: Search the Common Crawl Using Lambda Functions
   • URL: https://github.com/andresriancho/cc-lambda
   • AuthorName: Andres Riancho
   • Services: Lambda
5. tutorial:
   • Title: Large-scale graph mining with Spark
   • URL: https://towardsdatascience.com/large-scale-graph-mining-with-spark-750995050656
   • AuthorName: Win Suen
   • AuthorURL: https://github.com/wsuen/pygotham2018_graphmining

Tools & Applications:

1. tools & applications:

   • Title: Learning word vectors for 157 languages
   • URL: https://www.aclweb.org/anthology/L18-1550
   • AuthorName: Facebook AI Research
   • AuthorURL: https://fasttext.cc/docs/en/crawl-vectors.html

2. tools & applications:

   • Title: Dresden Web Table Corpus (DWTC)
   • URL: https://wwwdb.inf.tu-dresden.de/research-projects/dresden-web-table-corpus/
   • AuthorName: Database Systems Group Dresden
   • AuthorURL: https://wwwdb.inf.tu-dresden.de/

3. tools & applications:

   • Title: CCNet: Extracting high quality monolingual datasets from web crawl data
   • URL: https://arxiv.org/abs/1911.00359
   • AuthorName: Facebook AI Research
   • AuthorURL: https://github.com/facebookresearch/cc_net

Publication:

1. publication:

   • Title: Building a Web-Scale Dependency-Parsed Corpus from CommonCrawl
   • URL: https://arxiv.org/pdf/1710.01779.pdf
   • AuthorName: Alexander Panchenko, Eugen Ruppert, Stefano Faralli, Simone Paolo Ponzetto, Chris Biemann

2. publication:

   • Title: Using open data to predict market movements
   • URL: https://education.emc.com/content/dam/dell-emc/documents/en-us/2017KS_Ravinder-Using_Open_Data_to_Predict_Market_Movements.pdf
   • AuthorName: DELL EMC

3. publication:

   • Title: N-gram counts and language models from the Common Crawl
   • URL: http://www.lrec-conf.org/proceedings/lrec2014/pdf/1097_Paper.pdf
   • AuthorName: Christian Buck, Kenneth Heafield, Bas van Ooyen
   • AuthorURL: http://statmt.org/ngrams/

4. publication:

   • Title: Large-scale analysis of style injection by relative path overwrite
   • URL: https://doi.org/10.1145/3178876.3186090
   • AuthorName: Sajjad Arshad, Seyed Ali Mirheidari, Tobias Lauinger, Bruno Crispo, Engin Kirda, William Robertson

5. publication:

   • Title: Web Data Commons - RDFa, microdata, and microformat data sets
   • URL: http://webdatacommons.org/structureddata/
   • AuthorName: Christian Bizer, Robert Meusel, Anna Primpeli

6. publication:

   • Title: C4Corpus: Multilingual Web-Size Corpus with Free License
   • URL: http://www.lrec-conf.org/proceedings/lrec2016/pdf/388_Paper.pdf
   • AuthorName: Ivan Habernal, Omnia Zayed, Iryna Gurevych
   • AuthorURL: https://dkpro.github.io/dkpro-c4corpus/

7. publication:

   • Title: Of using Common Crawl to play Family Feud
   • URL: https://fulmicoton.com/posts/commoncrawl/
   • AuthorName: Paul Masurel

8. publication:

   • Title: Index fun
   • URL: https://psuter.net/2019/07/07/z-index
   • AuthorName: Philippe Suter

9. publication:

   • Title: Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures
   • URL: https://hal.inria.fr/hal-02148693
   • AuthorName: Pedro Javier Ortiz Suárez, Benoît Sagot, Laurent Romary
   • AuthorURL: https://oscar-corpus.com/

10. publication:

    • Title: Mapping languages: The Corpus of Global Language Use
    • URL: https://doi.org/10.1007/s10579-020-09489-2
    • AuthorName: Jonathan Dunn
    • AuthorURL: https://www.earthlings.io/

11. publication:

    • Title: CCAligned: A Massive collection of cross-lingual web-document pairs
    • URL: https://www.aclweb.org/anthology/2020.emnlp-main.480
    • AuthorName: Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, Philipp Koehn
    • AuthorURL: http://www.statmt.org/cc-aligned/

12. publication:

    • Title: CC-News-En: A large English news corpus
    • URL: https://doi.org/10.1145/3340531.3412762
    • AuthorName: Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R. Trippas, J. Shane Culpepper, Alistair Moffat

13. publication:

    • Title: Defending against neural fake news
    • URL: http://papers.nips.cc/paper/9106-defending-against-neural-fake-news.pdf
    • AuthorName: Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, et al
    • AuthorURL: https://rowanzellers.com/grover/

14. publication:

    • Title: On the impact of publicly available news and information transfer to financial markets
    • URL: https://arxiv.org/abs/2010.12002
    • AuthorName: Metod Jazbec, Barna Pásztor, Felix Faltings, Nino Antulov-Fantulin, Petter N. Kolm

15. publication:

    • Title: Language models are few-shot learners
    • URL: https://arxiv.org/abs/2005.14165
    • AuthorName: Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al