Skip to main content

Internet Archive Web Crawls

The Internet Archive discovers and captures web pages through many different web crawls.



rss RSS
play Play All

More right-solid
More right-solid
More right-solid
Worldwide Web Crawls
collection
624,708
ITEMS
11.2B
VIEWS
collection
eye 11.2
Survey Crawls
collection
98,189
ITEMS
6.8B
VIEWS
collection
eye 6.8
Topic: survey crawls
Live Web Proxy Crawls
collection
106,506
ITEMS
6.3B
VIEWS
collection
eye 6.3
collection
eye 1.5
collection
eye 1.4
Wide Crawl started April 2013
collection
25,035
ITEMS
981.4M
VIEWS
collection
eye 981.4
collection
eye 978.6
collection
eye 883.1
collection
eye 880.2
Wide Crawl Number 12 - started March, 14th 2015
collection
49,621
ITEMS
875.8M
VIEWS
collection
eye 875.8
Wide Crawl started June 2014
collection
45,341
ITEMS
859.2M
VIEWS
collection
eye 859.2
Wayback Indexes
collection
554
ITEMS
831.6M
VIEWS
collection
eye 831.6
Survey Crawl Number 6: Sep 11th, 2017 - running now
collection
13,723
ITEMS
758.7M
VIEWS
collection
eye 758.7
collection
eye 740.6
collection
eye 714.9
GDELT
collection
57,650
ITEMS
689.3M
VIEWS
collection
eye 689.3
Topics: GDELT, News
Wide Crawl started August 2013
collection
21,932
ITEMS
640.8M
VIEWS
collection
eye 640.8
Wide Crawl Number 13
collection
46,050
ITEMS
629.8M
VIEWS
collection
eye 629.8
Wide Crawl started January 2012
collection
30,373
ITEMS
564.3M
VIEWS
collection
eye 564.3
Wikipedia Outlinks March 2016
collection
27,594
ITEMS
552.1M
VIEWS
collection
eye 552.1
collection
eye 548.4
Survey Crawl Number 8
collection
8,699
ITEMS
536.9M
VIEWS
collection
eye 536.9
Survey Crawl Number 7
collection
6,605
ITEMS
521.5M
VIEWS
collection
eye 521.5
Wide Crawl started April 2012
collection
39,279
ITEMS
493.8M
VIEWS
collection
eye 493.8
collection
eye 471.5
Wide Crawl started February 2014
collection
9,806
ITEMS
386.4M
VIEWS
collection
eye 386.4
Wide Crawl started October 2010
collection
15,839
ITEMS
373M
VIEWS
collection
eye 373
Wide Crawl Started January 2013
collection
15,157
ITEMS
363.8M
VIEWS
collection
eye 363.8
Wide Crawl started September 2012
collection
22,423
ITEMS
359.8M
VIEWS
collection
eye 359.8
.com survey started January 2011
collection
2,535
ITEMS
351.1M
VIEWS
collection
eye 351.1
Topic: webcrawl
Wide Crawl started October 2011
collection
12,648
ITEMS
336.2M
VIEWS
collection
eye 336.2
Wide Crawl started March 2011
collection
8,528
ITEMS
314.9M
VIEWS
collection
eye 314.9
Around The World Crawl
collection
2,150
ITEMS
300.8M
VIEWS
collection
eye 300.8
Wikipedia Outlinks February 2012
collection
2,951
ITEMS
247.7M
VIEWS
collection
eye 247.7
collection
eye 232.5
International News Crawls
collection
6,965
ITEMS
159.7M
VIEWS
collection
eye 159.7
Shallow Crawls
collection
1,042
ITEMS
122.5M
VIEWS
collection
eye 122.5
Wikipedia Outlinks May 2011
collection
1,638
ITEMS
120.9M
VIEWS
collection
eye 120.9
Hackernews crawl number 0
collection
6,164
ITEMS
112.4M
VIEWS
collection
eye 112.4
Wayback CDX Shards
collection
1,214
ITEMS
104M
VIEWS
collection
eye 104
Wikipedia Outlinks July 2011
collection
1,011
ITEMS
84M
VIEWS
collection
eye 84
End of Term Web Crawls
collection
22,254
ITEMS
76.2M
VIEWS
by IA
collection
eye 76.2
Topics: web, end of term, US, federal government
Collections News Crawls v3
collection
4,825
ITEMS
73M
VIEWS
by ximm@archive.org
collection
eye 73
Topics: World news, US news, news
Survey Crawl Number 9
collection
502
ITEMS
69.5M
VIEWS
collection
eye 69.5
Geocities Closing Crawl
collection
149
ITEMS
60.1M
VIEWS
collection
eye 60.1
COM Survey Crawl 2009-2010
collection
729
ITEMS
55.8M
VIEWS
collection
eye 55.8
Youtube Videos
collection
489,161
ITEMS
53.3M
VIEWS
collection
eye 53.3
Topics: YouTube, Twitter, Video
Host Screen Captures
collection
17,295
ITEMS
52.1M
VIEWS
collection
eye 52.1
Shallow Crawl Started 2013
collection
544
ITEMS
49M
VIEWS
collection
eye 49
Shallow Crawl Started 2013
collection
252
ITEMS
48.7M
VIEWS
collection
eye 48.7
Live Web Proxy Crawls
web
eye 47.2M
favorite 2
comment 2
( 2 reviews )
survey_net00000
collection
300
ITEMS
41.5M
VIEWS
collection
eye 41.5
Topic: webcrawl
nsdlweb
collection
91
ITEMS
36.7M
VIEWS
collection
eye 36.7
Collections news crawls v2
collection
820
ITEMS
30.1M
VIEWS
by ximm@archive.org
collection
eye 30.1
International News Crawl started in September 2010
collection
1,083
ITEMS
29.4M
VIEWS
collection
eye 29.4
End of Term 2012 Web Crawls
collection
2,383
ITEMS
25.9M
VIEWS
collection
eye 25.9
Topics: end of term, US, Federal government, 2012, Obama
End of Term 2016 Web Crawls
collection
19,084
ITEMS
24.7M
VIEWS
collection
eye 24.7
Topics: end of term, federal government, 2016, president, congress, government data
ORG Survey Crawls
collection
191
ITEMS
24.5M
VIEWS
collection
eye 24.5
Live Web Proxy Crawls
web
eye 16M
favorite 5
comment 1
( 1 reviews ) Topic: crawldata
survey_net00001
collection
170
ITEMS
15.4M
VIEWS
collection
eye 15.4
Topics: webwidecrawl, net
Edu & Gov Crawl, June 2010
collection
704
ITEMS
15.1M
VIEWS
collection
eye 15.1
Topic: crawldata
Crawl Data
collection
32,956
ITEMS
14.9M
VIEWS
collection
eye 14.9
crawl_UNK
collection
32,949
ITEMS
14.9M
VIEWS
collection
eye 14.9
Shallow Crawl Started February 2013
collection
86
ITEMS
14.1M
VIEWS
collection
eye 14.1
Live Web Proxy Crawls
web
eye 13.4M
favorite 0
comment 0
collection
eye 12.7
Live Web Proxy Crawls
web
eye 12.1M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 12M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 11.1M
favorite 0
comment 0
Wide Crawl started September 2010
collection
332
ITEMS
10.8M
VIEWS
collection
eye 10.8
Live Web Proxy Crawls
web
eye 10.3M
favorite 0
comment 0
World Wars Crawl
collection
13
ITEMS
10.2M
VIEWS
collection
eye 10.2
2004 Election
collection
178
ITEMS
9.9M
VIEWS
collection
eye 9.9
End Of Term 2016 Pre-Inauguration Crawls
collection
4,693
ITEMS
9.2M
VIEWS
collection
eye 9.2
Topics: end of term, federal government, 2016, president, congress
YouTube 2007 Crawl
collection
73,847
ITEMS
8.8M
VIEWS
collection
eye 8.8
Government Web & Data Archive
collection
6,269
ITEMS
8.5M
VIEWS
collection
eye 8.5
Topics: government, data, federal, congress
Live Web Proxy Crawls
web
eye 8M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 8M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 8M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 8M
favorite 0
comment 0
Shallow Crawl Started November 2012
collection
137
ITEMS
7.9M
VIEWS
collection
eye 7.9
Live Web Proxy Crawls
web
eye 7.8M
favorite 0
comment 0
End of Term 2008 UNT Dot Gov Crawl
collection
281
ITEMS
7.7M
VIEWS
collection
eye 7.7
Live Web Proxy Crawls
web
eye 7.6M
favorite 0
comment 0
Hurricane Katrina
collection
112
ITEMS
7.5M
VIEWS
collection
eye 7.5
End Of Term 2016 UNT Crawls
collection
1,275
ITEMS
7.3M
VIEWS
collection
eye 7.3
Topics: end of term, federal government, 2016, president, congress, university of north texas
Topic: crawldata
Live Web Proxy Crawls
web
eye 6.6M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 6.5M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 6.4M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started January 2012
web
eye 5.9M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
web
eye 5.8M
favorite 0
comment 0
Wide Crawl started February 2014
web
eye 5.7M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
web
eye 5.7M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
web
eye 5.7M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
web
eye 5.7M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
web
eye 5.7M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 5.6M
favorite 0
comment 0
Wide Crawl started February 2014
web
eye 5.5M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
web
eye 5.4M
favorite 0
comment 0
Topic: crawldata
( 1 reviews ) Topic: crawldata
Live Web Proxy Crawls
web
eye 5.3M
favorite 1
comment 0
Live Web Proxy Crawls
web
eye 5.3M
favorite 0
comment 0
Topic: crawldata
2004 Indian Ocean earthquake and tsunami
collection
42
ITEMS
4.8M
VIEWS
collection
eye 4.8
Live Web Proxy Crawls
web
eye 4.8M
favorite 0
comment 0
Survey Crawl Number 8
web
eye 4.7M
favorite 0
comment 0
Topic: crawldata
Top 150 Crawl
collection
29
ITEMS
4.6M
VIEWS
collection
eye 4.6
Live Web Proxy Crawls
web
eye 4.6M
favorite 0
comment 0
UNT Web
collection
35
ITEMS
4.6M
VIEWS
collection
eye 4.6
Topics: UNT, web, texas, eot
Live Web Proxy Crawls
web
eye 4.4M
favorite 0
comment 0
Topic: crawldata
UK Government Site Crawl
collection
106
ITEMS
4.4M
VIEWS
collection
eye 4.4
Topic: crawldata
Live Web Proxy Crawls
web
eye 4.1M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
web
eye 4.1M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
web
eye 4.1M
favorite 0
comment 0
Topic: crawldata
Wayback Robots Crawl
collection
129
ITEMS
4.1M
VIEWS
collection
eye 4.1
Live Web Proxy Crawls
web
eye 4.1M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.9M
favorite 0
comment 0
Topic: crawldata
End Of Term 2016 Library of Congress Crawls
collection
3,892
ITEMS
3.8M
VIEWS
collection
eye 3.8
Topics: end of term, federal government, 2016, president, congress, library of congress, web, data, library...
Live Web Proxy Crawls
web
eye 3.7M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.6M
favorite 0
comment 0
Topic: crawldata
End of Term 2016 Post-Inauguration Crawls
collection
5,308
ITEMS
3.6M
VIEWS
collection
eye 3.6
Topics: end of term, federal government, 2016, president, congress
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.5M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started June 2014
web
eye 3.5M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started June 2014
web
eye 3.5M
favorite 0
comment 0
Topic: crawldata
Topic: crawldata
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.4M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.3M
favorite 0
comment 0
Topic: crawldata
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.2M
favorite 0
comment 0
Live Web Proxy Crawls
web
eye 3.2M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Topic: crawldata
Topic: crawldata
Topic: crawldata
Topic: crawldata
Topic: crawldata
Topic: crawldata
Live Web Proxy Crawls
web
eye 3M
favorite 0
comment 0
Topic: crawldata
Topic: crawldata
Topic: crawldata
Topic: crawldata
Live Web Proxy Crawls
web
eye 2.9M
favorite 0
comment 0