COLLECTED BY

Organization: Internet Archive

The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.

Collection: Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What’s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

TIMESTAMPS

The Wayback Machine - https://web.archive.org/web/20110824230223/http://www.politico.com/aboutus/

Find stories by:

Author Or
Date Or
Keywords

POLITICO aboutus

Submit a question to debate candidates

About Us

POLITICO

1100 Wilson BLVD, Suite 610

Arlington, VA, 22209

Submit news releases and tips

Executive

Publisher: Robert L. Allbritton

President & CEO: Frederick J. Ryan Jr.

Chief Operating Officer: Kim Kingsley

Editorial

Editor-in-Chief: John F. Harris

Executive Editor: Jim VandeHei

Managing Editor: Bill Nichols

Managing Editor, Online: Danielle Jones

Chief Deputy Managing Editor: Craig Gordon

Deputy Managing Editor: Bill Hamilton

Assistant Managing Editor, Partnerships: Beth Frerking

Executive Director of Digital Innovation: Hemal Jhaveri

National Politics Editor: Charles Mahtesian

Congressional Editor: Martin Kady II

Senior Editors: Gregg Birnbaum; David Cohen; G. Robert Hillman; Craig Howie; Russ Walker

Arena Editor: David Mark

Opinion Editor: Allison Silver

White House Bureau Chief: Rachel Smolkin

Congressional Bureau Chief: Laura McGann

Deputy Politics Editors: Edward-Isaac Dovere; Sara Libby

44 Editor: Matt Negrin

Deputy Web Editor: Dianna Heitz

CLICK Editor: Caitlin McDevitt

Assistant Editors: Erika Lovley; MJ Lee; Jedd Rosche

Contributing Editors: Andie Coller; Andrew Glass

Chief White House Correspondent: Mike Allen

Senior Political Reporters: Maggie Haberman; Jonathan Martin; Ben Smith

Senior Congressional Reporters: Jonathan Allen; John Bresnahan; David Rogers

Senior White House Reporter: Glenn Thrush

Chief Investigative Reporter: Kenneth P. Vogel

Chief Political Columnist: Roger Simon

Reporters: Molly Ball,; Josh Boak,; Carrie Budoff Brown,; Alexander Burns,; David Catanese,; Marin Cogan,; Jennifer Epstein,; Reid Epstein,; Patrick Gavin,; Josh Gerstein,; Keach Hagey,; Dan Hirschhorn,; James Hohmann,; Charles Hoskinson,; Alex Isenstadt,; Dave Levinthal,; Seung Min Kim,; Tim Mak,; Kendra Marr,; Julie Mason,; Anna Palmer,; Amie Parnes,; Abby Phillip,; Manu Raju,; Emily Schultheis,; Jake Sherman,; Juana Summers,; Karin Tanabe,; Ben White,; Joseph Williams,; Scott Wong

Researcher: Zachary Abrahamson

Editorial Assistants: Lucy McCalmont; Kevin Robillard; Byron Tau

Director of Administration: Elizabeth Breckenridge

Executive Assistant: Anna Bacon

Editorial Cartoonist: Matt Wuerker

Copy Desk Chief: Judy Shapleigh

Copy Editors: Maggie Chan; Andy Goodwin; Puja Murgai; Sushant Sagar; Robin Turner

Design Director: Sarah Lauren Bell

Designers: Janeen Jones; Bill Kuchman; Kristen Long

Director of Photography: Michael Schwartz

Staff Photographers: John Shinkle; Jay Westcott

Director of Search and Analytics: Mitchell Schuler

Senior Web Producer: Anita Ford

Web Producers: Alex Byers; Bryan Doyle; Burgess Everett; Zack Hale; Emily Howell; Bridget Mulcahy; Leigh Munsil; Naira Ruiz; Vivyan Tran; Austin Wright

POLITICO Pro Editorial

Editor-in-Chief: Tim Grieve

Deputy Managing Editors: Dan Berman; LaRonda Peterson

Health Care Editor: David Nather

Technology Editor: Elizabeth Wasserman

Deputy Energy Editor: Bob King

Deputy Technology Editor: Mike Zapler

Senior Energy and Environment Reporter: Darren Samuelsohn

Senior Technology Reporter: Kim Hart

Energy Reporters: Robin Bravender; Talia Buford; Darius Dixon; Darren Goode; Alex Guillén; Patrick Reis

Health Care Reporters: Matt DoBias; J. Lester Feder; Jennifer Haberkorn; Jason Millman; Kate Nocera; Brett Norman

Technology Reporters: Brooks Boliek; Eliza Krigman; Jennifer Martinez; David Saleh Rauf; Tony Romm

Silicon Valley Correspondent: Michelle Quinn

Copy Editors: Brianna Labuskes; Abby McIntyre

Web Producers: Brittany Hackett; Julia Haslanger; Jess Kamen; Kathryn Smith

Media & Special Projects

Executive Director of Media & Special Projects: Sara Olson

Senior Project Manager: Katherine Lehr

Media Coordinator: Lauren Pulte

Chief Video Producer: Matthew Sobocinski

Video Producer: Julio Negron

Assistant Editor, Video: Alexander Trowbridge

Special Projects Assistant: Katie Maloney

Technology

Chief Technology Officer: Ryan P. Mannion

Director of Technical Operations: Mike Devich

Director of Web Development: Kris Jones

Director of Mobile Development: Mark Gerl

Director of User Experience: Chris Buddie

Senior Technical Project Manager: Jen Dreyer

Development Team Lead: Liz Kaplan

Mobile Applications Architect, Team Lead: Charles Lee

Senior Web Designer, Team Lead: Jared Cunha

Senior Web Developers: Han Ah-Sue; Kevin Benore; Donna Caudill; Ray Gurganus; Patrick Sheridan

Senior Mobile Developers: Jerome Davis; Andrew Kinnie

Senior Systems Administrators: Chris Behm; Paul Frederiksen

Flash Developer: Christian Guirreri

Senior Web Designers: Nathan Gomez; Michael Marcialis

Ad Management Designer: Lee Umali

Business & Advertising

Chief Revenue Officer: Roy Schwartz

Vice President and Associate Publisher: Michael B. McGrath

Vice President of Events: Beth Lester

Executive Director of Business Development, POLITICO Pro: Miki King

Business Manager: Jennifer Hurley

National Advertising Directors: Aaron Finley; Jacquelyn Humenik; Erin Matticola; Adam Prather; Jack Smith

Director of Business Development: Michael Schoenbrun

Director of Strategic Accounts: Shan Wade

Account Managers: Megan Dorward; Cally Stolbach

Retail Account Executive: Kary Jones

Classifieds Manager: Matt Donohue

Business Development Executives, POLITICO Pro: Bobby Moran; Daryl A. Muller

Account Managers, POLITICO Pro: Lauren Englander; Amanda Gross; Simone A. White

Customer Relations Associate, POLITICO Pro: Jessie Markley

Circulation Director: John Lipp

Advertising Operations Manager: Walt Houseknecht

Advertising Operations Specialist: Das Rios

Events Manager: Anne Lyons

Events Coordinators: Ashley Summers; Alexis Williams

Accounting Manager: Suhani Sanwar

Payroll/Benefits Specialist: Teresa Escobar

Graphic Designer: Anand Gahlot

Staff Accountant: Richard Cardona

Sales and Marketing Analyst: Pat Diggins