Common Crawl's "Right to Learn" AI Event

Common Crawl Foundation hosted an event in New York in April for a select group of leaders in AI, technology, media and content to discuss AI & The Right to Learn on an Open Internet.

SutherlandGold Co-Founder Lesley Gold is an advisor to Common Crawl and helped craft the event. Common Crawl is the “Large” in the Large Language Models (LLMs) that are critical to fueling AI advances and breakthroughs, according to CEO Rich Skrenta. The event was titled, “AI & The Right to Learn on an Open Internet”.

Crawling the Internet to assemble all of mankind’s information to help future generations learn should be a common goal, not something that is litigated or fought over. The conference was intended to foster an open dialogue between stakeholders about how to achieve this common goal of supporting a right to learn on an open Internet.

Sponsors of the conference included Kearney: Global Management Consulting Firm, Tola Capital, and Computer & Communications Industry Association (CCIA).

The one-day event held at the Craig Newmark Graduate School of Journalism at The City University of New York (CUNY) featured opening remarks, 6 firestarter mini-sessions, 4 panel discussions, demo time, networking opportunities, and more.

Key Metrics on Common Crawl:

• Vision that Web crawls should be open, not kept in walled gardens

• 250 billion pages spanning 17 years

• Cited in 10,000+ research papers

• A critical resource for AI research and development

• Common Crawl covers 200+ languages, 10 petabytes

• Primary training corpus for nearly every LLM

• The “Linux” of open source training data

Skrenta said Common Crawl is considering hosting a similar event in Europe later this year, possibly in Paris, which is emerging as a new AI hub.

“But there’s so much going on in the AI space - you could go do something pretty much every day somewhere around the world,” Skrenta said. “We want to take the temperature of the room wherever we go and make sure that we have something unique to say about really using all the web’s information to create real learning for the world.”

Skrenta said he felt like Professor Jeff Jarvis's wrap-up at the very end of the day where he ran around the room Oprah-style asking the audience questions was the perfect finale to a day of open discussion.

“I think letting everyone get a chance to raise issues and ask questions of anyone in the room and hear their commentary and thoughts really opened things up to the floor,” Skrenta said. “I think that was the perfect way to end a day that was really about driving discussion.”