The Open Syllabus collection contains WARC files from a mid-2021 crawl of about 50 million unique seed URLs extracted from the Open Syllabus version 2.6 dataset and their page requisites. The bulk of the seed URLs are from ".com", ".org", ".edu", and ".uk" TLDs.
Crawl Summary
- Crawl start: 2021-04-12
- Crawl end: 2021-09-05
- Seed URLs: 49,735,419
- Archived URLs: 338,690,414
- Collection Size: 25 TB
- Crawler: Heritrix/3.3.0-hq1-SNAPSHOT-2015-03-16T18:09:23Z
- Crawl depth: maxHops=0
Seed Summary
- Unique URLs: 49,735,419
- Unique Canonical URLs: 48,956,395
- Unique Hosts: 984,223
- IPv4 Addresses: 3,328
- Unique TLDs: 21,761
- Unique IANA Valid TLDs: 739
- Wayback Machine URLs*: 6,568,213
* NOTE: More than 13% URLs in the dataset point to Wayback Machine!
We're Social...for Statheads
Every Sports Reference Social Media Account
Site Last Updated: Monday, April 19, 6:43AM
Question, Comment, Feedback, or Correction?
Are you a Stathead, too? Subscribe to our Free Newsletter
This Month in Sports Reference
Find out when we add a feature or make a change
Do you have a sports website? Or write about sports? We have tools and resources that can help you use sports data. Find out more.