Common Crawl maintains a free, open repository of web crawl data that can be used by anyone. It provides wholesale extraction, transformation and analysis of open web data accessible to researchers.

Common Crawl

Common Crawl Foundation | 918 followers on LinkedIn. Common Crawl maintains a free, open repository of web crawl data that can be used by anyone. | The Common Crawl Foundation is a California 501(c)(3) registered non-profit founded by Gil Elbaz with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable.

Our vision is of a truly open web that allows open access to information and enables greater innovation in research, business and education. We level the playing field by making wholesale extraction, transformation and analysis of web data cheap and easy.

Executive Director at Common Crawl Foundation

Rich Skrenta

Product Manager

Wayne Yamamoto

Leading SEO & Web Strategy Expert | Developmental Leader

Stephen Burns

Engineer, Founder, Advisor, Strategic Angel

Sarmeesha Reddy

Common Crawl Foundation

We are pleased to announce a new index and query api system for Common Crawl! We're excited that the Common Crawl community is embracing this new feature and have already added January and February 2015 datasets (that's 300+ TB). Going forward, each month's crawl will be accompanied by a new index. Check out the full announcement on our blog. https://lnkd.in/bJn4rp2

Last week I had the opportunity to present a series of talks on "Harnessing Common Crawl at Scale" with Pedro Ortiz Suarez . We presented a talk at the The Alan Turing Institute 's NLP Special Interest Group, and another in collaboration with Valyu at UCL . Here's the Common Crawl Foundation 's blog post on these talks: 🔗 https://lnkd.in/gJNJc9Q2 Many thanks to Robert Blackwell, PhD and Anthony Hills from the Turing Institute; Hirsh Pithadia , Harvey Yorke , and Hendrik van der Sande from Valyu; and Prof. Philip Treleaven from UCL.

Thom Vaughan

Great talks on our "Open Data, Research and Web Archiving in the Age of AI and LLMs" event today and thanks to everyone who attended and especially Thom Vaughan and Pedro Ortiz Suarez from Common Crawl Foundation for presenting. Stay tuned for our next event! #OpenData #WebCrawling Hirsh Pithadia

Valyu

Common Crawl Foundation and Constellation Network Announce Partnership to Bridge Blockchain and AI Rich Skrenta Benjamin Jorgensen https://lnkd.in/d6zp5VvT

aiTech Trend

I’m pleased to share the release of Common Crawl Foundation 's September crawl and corresponding Web Graph release. 🤖 The September crawl contains 2.8 billion web pages (410 TiB of uncompressed content), and the corresponding Web Graph includes 306.5 million nodes and 2.6 billion edges at the host level, and 95.4 million nodes and 1.7 billion edges at the domain level. More information can be found in these blog posts: 🔗 September 2024 Crawl Archive https://lnkd.in/gBhmQsYP 🔗 Web Graph July-August-September 2024 https://lnkd.in/gNU2RR3p As always, questions or comments are welcome.

I’m thrilled to announce the Alliance for Responsible Data Collection (ARDC) will be presenting our position paper, “Technical and Governance Guidelines for Responsible Data Collection”, at the IAB AI-Controls Workshop in Washington, DC this week! 🌍📊 Our focus: creating a trusted framework for responsible data collection that supports transparency, choice and accountability for varying downstream uses. 🛠️ This is a pivotal time for the future of data collection and all who rely on responsibly sourced public internet data. https://lnkd.in/gMWmWs88 Stay tuned for more updates! 💡 #ARDC #ResponsibleData #AI #DataCollection #BestPractices #Transparency #IABWorkshop #AI #responsiblysourceddata

COMM

ORG:COMMONCRAWL

Common Crawl