Skip to main content

Internet Archive Web Crawls

The Internet Archive discovers and captures web pages through many different web crawls.



rss RSS
play Play All

More right-solid
More right-solid
More right-solid
More right-solid
SHOW DETAILS
up-solid down-solid
Worldwide Web Crawls
collection
624,708
ITEMS
11.2B
VIEWS
collection
eye 11.2
Survey Crawls
collection
98,189
ITEMS
6.8B
VIEWS
collection
eye 6.8
Topic: survey crawls
Live Web Proxy Crawls
collection
106,506
ITEMS
6.3B
VIEWS
collection
eye 6.3
Wide Crawl Number 14 - Started Mar 4th, 2016 - Ended Sep 15th, 2016
collection
71,768
ITEMS
1.5B
VIEWS
collection
eye 1.5
Survey Crawl Number 0 - Started May 18th, 2013 - Ended May 15, 2014
collection
16,282
ITEMS
1.4B
VIEWS
collection
eye 1.4
Wide Crawl started April 2013
collection
25,035
ITEMS
981.4M
VIEWS
collection
eye 981.4
Survey Crawl Number 2 - Started Dec 17th, 2014 - Ended Jul 31st, 2015
collection
13,890
ITEMS
978.6M
VIEWS
collection
eye 978.6
Wide Crawl Number 15: Started Oct 1st, 2016 - Ended May 8th, 2017
collection
38,827
ITEMS
883.1M
VIEWS
collection
eye 883.1
Wide Crawl Number 17: Started August 3rd, 2018 - Still running
collection
69,688
ITEMS
880.2M
VIEWS
collection
eye 880.2
Wide Crawl Number 12 - started March, 14th 2015
collection
49,621
ITEMS
875.8M
VIEWS
collection
eye 875.8
Wide Crawl started June 2014
collection
45,341
ITEMS
859.2M
VIEWS
collection
eye 859.2
Wayback Indexes
collection
554
ITEMS
831.6M
VIEWS
collection
eye 831.6
Survey Crawl Number 6: Sep 11th, 2017 - running now
collection
13,723
ITEMS
758.7M
VIEWS
collection
eye 758.7
Survey Crawl Number 4 - Started Jan 9th, 2016 - Ended Feb 25th, 2016
collection
12,622
ITEMS
740.6M
VIEWS
collection
eye 740.6
Wide Crawl Number 16: Started June 3rd, 2017 - Still running
collection
83,809
ITEMS
714.9M
VIEWS
collection
eye 714.9
GDELT
collection
57,650
ITEMS
689.3M
VIEWS
collection
eye 689.3
Topics: GDELT, News
Wide Crawl started August 2013
collection
21,932
ITEMS
640.8M
VIEWS
collection
eye 640.8
Wide Crawl Number 13
collection
46,050
ITEMS
629.8M
VIEWS
collection
eye 629.8
Wide Crawl started January 2012
collection
30,373
ITEMS
564.3M
VIEWS
collection
eye 564.3
Wikipedia Outlinks March 2016
collection
27,594
ITEMS
552.1M
VIEWS
collection
eye 552.1
Survey Crawl Number 3 - Started Aug 1st, 2015 - Ended Feb 11th, 2016
collection
10,137
ITEMS
548.4M
VIEWS
collection
eye 548.4
Survey Crawl Number 8
collection
8,699
ITEMS
536.9M
VIEWS
collection
eye 536.9
Survey Crawl Number 7
collection
6,605
ITEMS
521.5M
VIEWS
collection
eye 521.5
Wide Crawl started April 2012
collection
39,279
ITEMS
493.8M
VIEWS
collection
eye 493.8
Survey Crawl Number 1 - Started May 17th, 2014 - Ended Dec 10th, 2014
collection
6,909
ITEMS
471.5M
VIEWS
collection
eye 471.5
Wide Crawl started February 2014
collection
9,806
ITEMS
386.4M
VIEWS
collection
eye 386.4
Wide Crawl started October 2010
collection
15,839
ITEMS
373M
VIEWS
collection
eye 373
Wide Crawl Started January 2013
collection
15,157
ITEMS
363.8M
VIEWS
collection
eye 363.8
Wide Crawl started September 2012
collection
22,423
ITEMS
359.8M
VIEWS
collection
eye 359.8
.com survey started January 2011
collection
2,535
ITEMS
351.1M
VIEWS
collection
eye 351.1
Topic: webcrawl
Wide Crawl started October 2011
collection
12,648
ITEMS
336.2M
VIEWS
collection
eye 336.2
Wide Crawl started March 2011
collection
8,528
ITEMS
314.9M
VIEWS
collection
eye 314.9
Around The World Crawl
collection
2,150
ITEMS
300.8M
VIEWS
collection
eye 300.8
Wikipedia Outlinks February 2012
collection
2,951
ITEMS
247.7M
VIEWS
collection
eye 247.7
Survey Crawl Number 5: Oct 21st, 2016 to Sep 10th, 2017
collection
3,888
ITEMS
232.5M
VIEWS
collection
eye 232.5
International News Crawls
collection
6,965
ITEMS
159.7M
VIEWS
collection
eye 159.7
Shallow Crawls
collection
1,042
ITEMS
122.5M
VIEWS
collection
eye 122.5
Wikipedia Outlinks May 2011
collection
1,638
ITEMS
120.9M
VIEWS
collection
eye 120.9
Hackernews crawl number 0
collection
6,164
ITEMS
112.4M
VIEWS
collection
eye 112.4
Wayback CDX Shards
collection
1,214
ITEMS
104M
VIEWS
collection
eye 104
Wikipedia Outlinks July 2011
collection
1,011
ITEMS
84M
VIEWS
collection
eye 84
End of Term Web Crawls
collection
22,254
ITEMS
76.2M
VIEWS
by IA
collection
eye 76.2
Topics: web, end of term, US, federal government
Collections News Crawls v3
collection
4,825
ITEMS
73M
VIEWS
by ximm@archive.org
collection
eye 73
Topics: World news, US news, news
Survey Crawl Number 9
collection
502
ITEMS
69.5M
VIEWS
collection
eye 69.5
Geocities Closing Crawl
collection
149
ITEMS
60.1M
VIEWS
collection
eye 60.1
COM Survey Crawl 2009-2010
collection
729
ITEMS
55.8M
VIEWS
collection
eye 55.8
Youtube Videos
collection
489,161
ITEMS
53.3M
VIEWS
collection
eye 53.3
Topics: YouTube, Twitter, Video
Host Screen Captures
collection
17,295
ITEMS
52.1M
VIEWS
collection
eye 52.1
Shallow Crawl Started 2013
collection
544
ITEMS
49M
VIEWS
collection
eye 49
Shallow Crawl Started 2013
collection
252
ITEMS
48.7M
VIEWS
collection
eye 48.7
Live Web Proxy Crawls
liveweb-20190803151256
web
eye 47.2M
favorite 2
comment 2
( 2 reviews )
survey_net00000
collection
300
ITEMS
41.5M
VIEWS
collection
eye 41.5
Topic: webcrawl
nsdlweb
collection
91
ITEMS
36.7M
VIEWS
collection
eye 36.7
Collections news crawls v2
collection
820
ITEMS
30.1M
VIEWS
by ximm@archive.org
collection
eye 30.1
International News Crawl started in September 2010
collection
1,083
ITEMS
29.4M
VIEWS
collection
eye 29.4
End of Term 2012 Web Crawls
collection
2,383
ITEMS
25.9M
VIEWS
collection
eye 25.9
Topics: end of term, US, Federal government, 2012, Obama
End of Term 2016 Web Crawls
collection
19,084
ITEMS
24.7M
VIEWS
collection
eye 24.7
Topics: end of term, federal government, 2016, president, congress, government data
ORG Survey Crawls
collection
191
ITEMS
24.5M
VIEWS
collection
eye 24.5
Live Web Proxy Crawls
Liveweb Capture 2011-03-27T22:10:09PDT to 2011-03-28T05:27:05PDT
by Internet Archive
web
eye 16M
favorite 5
comment 1
( 1 reviews ) Topic: crawldata
survey_net00001
collection
170
ITEMS
15.4M
VIEWS
collection
eye 15.4
Topics: webwidecrawl, net
Edu & Gov Crawl, June 2010
collection
704
ITEMS
15.1M
VIEWS
collection
eye 15.1
Topic: crawldata
Crawl Data
collection
32,956
ITEMS
14.9M
VIEWS
collection
eye 14.9
crawl_UNK
collection
32,949
ITEMS
14.9M
VIEWS
collection
eye 14.9
Shallow Crawl Started February 2013
collection
86
ITEMS
14.1M
VIEWS
collection
eye 14.1
Live Web Proxy Crawls
liveweb-20181028210043
web
eye 13.4M
favorite 0
comment 0
End of Term 2008 California Digital Library Crawl
collection
415
ITEMS
12.7M
VIEWS
collection
eye 12.7
Live Web Proxy Crawls
liveweb-20181211002001
web
eye 12.1M
favorite 0
comment 0
Live Web Proxy Crawls
liveweb-20181030213611
web
eye 12M
favorite 0
comment 0
Live Web Proxy Crawls
liveweb-20161130152241
web
eye 11.1M
favorite 0
comment 0
Wide Crawl started September 2010
collection
332
ITEMS
10.8M
VIEWS
collection
eye 10.8
Live Web Proxy Crawls
liveweb-20190808115031
web
eye 10.3M
favorite 0
comment 0
World Wars Crawl
collection
13
ITEMS
10.2M
VIEWS
collection
eye 10.2
2004 Election
collection
178
ITEMS
9.9M
VIEWS
collection
eye 9.9
End Of Term 2016 Pre-Inauguration Crawls
collection
4,693
ITEMS
9.2M
VIEWS
collection
eye 9.2
Topics: end of term, federal government, 2016, president, congress
YouTube 2007 Crawl
collection
73,847
ITEMS
8.8M
VIEWS
collection
eye 8.8
Government Web & Data Archive
collection
6,269
ITEMS
8.5M
VIEWS
collection
eye 8.5
Topics: government, data, federal, congress
Live Web Proxy Crawls
liveweb-20170906090536
web
eye 8M
favorite 0
comment 0
Live Web Proxy Crawls
liveweb-20170906072959
web
eye 8M
favorite 0
comment 0
Live Web Proxy Crawls
liveweb-20180411074817
web
eye 8M
favorite 0
comment 0
Live Web Proxy Crawls
liveweb-20180411085101
web
eye 8M
favorite 0
comment 0
Shallow Crawl Started November 2012
collection
137
ITEMS
7.9M
VIEWS
collection
eye 7.9
Live Web Proxy Crawls
liveweb-20190624183849
web
eye 7.8M
favorite 0
comment 0
End of Term 2008 UNT Dot Gov Crawl
collection
281
ITEMS
7.7M
VIEWS
collection
eye 7.7
Live Web Proxy Crawls
liveweb-20180413122557
web
eye 7.6M
favorite 0
comment 0
Hurricane Katrina
collection
112
ITEMS
7.5M
VIEWS
collection
eye 7.5
End Of Term 2016 UNT Crawls
collection
1,275
ITEMS
7.3M
VIEWS
collection
eye 7.3
Topics: end of term, federal government, 2016, president, congress, university of north texas
World Wars Crawl
Crawldata from Internet Archive from 2008-08-08T00:17:40PDT to 2008-06-26T05:29:33PDT
web
eye 7.2M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
liveweb-20171009123355
web
eye 6.6M
favorite 0
comment 0
Live Web Proxy Crawls
liveweb-20180617131556
web
eye 6.5M
favorite 0
comment 0
Live Web Proxy Crawls
liveweb-20200405053634
web
eye 6.4M
favorite 0
comment 0
Wide Crawl Number 16: Started June 3rd, 2017 - Still running
Webwide Crawldata 2018-02-12T21:42:38PST to 2018-02-12T15:20:34PST
by Internet Archive
web
eye 6.2M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started January 2012
Webwide Crawldata 2012-01-17T08:02:53PST to 2012-01-17T01:16:20PST
by Internet Archive
web
eye 5.9M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
liveweb-20191203123152
web
eye 5.8M
favorite 0
comment 0
Wide Crawl started February 2014
Webwide Crawldata 2014-02-19T01:09:37PST to 2014-02-18T21:33:27PST
by Internet Archive
web
eye 5.7M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
Webwide Crawldata 2014-02-19T05:20:19PST to 2014-02-19T01:54:33PST
by Internet Archive
web
eye 5.7M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
Webwide Crawldata 2014-02-18T17:01:58PST to 2014-02-18T13:14:06PST
by Internet Archive
web
eye 5.7M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
Webwide Crawldata 2014-02-19T09:49:01PST to 2014-02-19T06:07:15PST
by Internet Archive
web
eye 5.7M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
liveweb-20150202030711
web
eye 5.7M
favorite 0
comment 0
Live Web Proxy Crawls
liveweb-20180413214739
web
eye 5.6M
favorite 0
comment 0
Wide Crawl started February 2014
Webwide Crawldata 2014-02-19T07:58:38PST to 2014-02-19T05:13:46PST
by Internet Archive
web
eye 5.5M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2011-03-28T12:43:47PDT to 2011-03-28T16:56:17PDT
by Internet Archive
web
eye 5.4M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2002-11-01T06:23:33PDT to 2002-11-19T23:24:07PDT
web
eye 5.4M
favorite 0
comment 1
( 1 reviews ) Topic: crawldata
Live Web Proxy Crawls
liveweb-20151110214543
web
eye 5.3M
favorite 1
comment 0
Live Web Proxy Crawls
Liveweb Capture 2011-05-08T07:07:52PDT to 2011-05-08T08:00:29PDT
by Internet Archive
web
eye 5.3M
favorite 0
comment 0
Topic: crawldata
2004 Indian Ocean earthquake and tsunami
collection
42
ITEMS
4.8M
VIEWS
collection
eye 4.8
Live Web Proxy Crawls
liveweb-20170830054437
web
eye 4.8M
favorite 0
comment 0
Survey Crawl Number 8
Survey Webwide Crawldata 2018-12-14T21:00:08PST to 2018-12-14T22:34:06PST
by Internet Archive
web
eye 4.7M
favorite 0
comment 0
Topic: crawldata
Top 150 Crawl
collection
29
ITEMS
4.6M
VIEWS
collection
eye 4.6
Live Web Proxy Crawls
liveweb-20200121124702
web
eye 4.6M
favorite 0
comment 0
UNT Web
collection
35
ITEMS
4.6M
VIEWS
collection
eye 4.6
Topics: UNT, web, texas, eot
Live Web Proxy Crawls
Liveweb Capture 2011-03-29T00:12:24PDT to 2011-03-29T07:24:41PDT
by Internet Archive
web
eye 4.4M
favorite 0
comment 0
Topic: crawldata
UK Government Site Crawl
collection
106
ITEMS
4.4M
VIEWS
collection
eye 4.4
Survey Crawl Number 6: Sep 11th, 2017 - running now
Survey Webwide Crawldata 2017-10-12T08:48:34PDT to 2017-10-12T01:56:31PDT
by Internet Archive
web
eye 4.2M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2011-05-20T00:54:34PDT to 2011-05-20T04:55:47PDT
by Internet Archive
web
eye 4.1M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
Webwide Crawldata 2014-02-19T08:18:23PST to 2014-02-19T04:21:37PST
by Internet Archive
web
eye 4.1M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started February 2014
Webwide Crawldata 2014-02-18T22:58:46PST to 2014-02-18T19:25:19PST
by Internet Archive
web
eye 4.1M
favorite 0
comment 0
Topic: crawldata
Wayback Robots Crawl
collection
129
ITEMS
4.1M
VIEWS
collection
eye 4.1
Live Web Proxy Crawls
Liveweb Capture 2011-05-19T17:19:06PDT to 2011-05-19T17:46:28PDT
by Internet Archive
web
eye 4.1M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2011-03-30T03:11:49PDT to 2011-03-30T10:24:04PDT
by Internet Archive
web
eye 3.9M
favorite 0
comment 0
Topic: crawldata
End Of Term 2016 Library of Congress Crawls
collection
3,892
ITEMS
3.8M
VIEWS
collection
eye 3.8
Topics: end of term, federal government, 2016, president, congress, library of congress, web, data, library...
Live Web Proxy Crawls
Liveweb Capture 2013-01-09T09:11:28 UTC to 2013-01-09T20:54:17 UTC
by Internet Archive
web
eye 3.7M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2013-01-09T16:00:53 UTC to 2013-01-10T01:01:51 UTC
by Internet Archive
web
eye 3.6M
favorite 0
comment 0
Topic: crawldata
End of Term 2016 Post-Inauguration Crawls
collection
5,308
ITEMS
3.6M
VIEWS
collection
eye 3.6
Topics: end of term, federal government, 2016, president, congress
Wayback Indexes
Crawldata from Internet Archive from 2003-09-26T12:13:03PDT to 2003-12-10T23:25:51PDT
web
eye 3.5M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2011-03-29T14:27:39PDT to 2011-03-29T20:01:59PDT
by Internet Archive
web
eye 3.5M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started June 2014
Webwide Crawldata 2014-07-10T07:24:15PDT to 2014-07-10T01:45:52PDT
by Internet Archive
web
eye 3.5M
favorite 0
comment 0
Topic: crawldata
Wide Crawl started June 2014
Webwide Crawldata 2014-07-10T06:43:41PDT to 2014-07-10T01:23:01PDT
by Internet Archive
web
eye 3.5M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2004-09-18T12:46:38PDT to 2005-05-05T09:34:36PDT
web
eye 3.4M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2003-11-16T11:09:15PDT to 2003-06-17T19:02:43PDT
web
eye 3.4M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2013-03-29T09:54:55 UTC to 2013-03-29T13:00:26 UTC
by Internet Archive
web
eye 3.4M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2013-10-14T07:53:13 UTC to 2013-10-18T09:13:35 UTC
by Internet Archive
web
eye 3.3M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2005-08-14T21:06:52PDT to 2005-11-20T23:12:32PDT
web
eye 3.3M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
liveweb-20190803130243
web
eye 3.2M
favorite 0
comment 0
Live Web Proxy Crawls
Liveweb Capture 2011-03-31T04:44:35PDT to 2011-03-31T13:13:56PDT
by Internet Archive
web
eye 3.2M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2013-03-29T09:35:58 UTC to 2013-03-29T15:03:34 UTC
by Internet Archive
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2003-11-27T00:45:40PDT to 2003-03-29T12:46:47PDT
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2011-05-20T12:11:05PDT to 2011-05-20T16:12:31PDT
by Internet Archive
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
Liveweb Capture 2013-07-11T00:40:27 UTC to 2013-07-11T05:26:31 UTC
by Internet Archive
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2005-05-04T22:04:03PDT to 2005-10-25T16:31:50PDT
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2003-03-29T18:35:25PDT to 2003-11-16T08:00:57PDT
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2003-06-16T20:25:23PDT to 2003-11-26T17:48:35PDT
web
eye 3.1M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2004-08-02T08:33:20PDT to 2005-06-02T08:07:17PDT
web
eye 3M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2005-03-08T20:18:06PDT to 2005-09-27T05:06:47PDT
web
eye 3M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2005-03-05T13:09:57PDT to 2005-10-29T13:54:47PDT
web
eye 3M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
liveweb-20190916013320
web
eye 3M
favorite 0
comment 0
Wayback Indexes
Crawldata from Internet Archive from 2003-06-19T05:30:16PDT to 2003-11-26T23:13:24PDT
web
eye 3M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2006-05-03T07:47:56PDT to 2004-09-17T05:50:57PDT
web
eye 2.9M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2003-06-17T21:04:33PDT to 2003-11-29T14:39:46PDT
web
eye 2.9M
favorite 0
comment 0
Topic: crawldata
Wayback Indexes
Crawldata from Internet Archive from 2004-04-21T08:36:57PDT to 2004-12-03T12:31:33PDT
web
eye 2.9M
favorite 0
comment 0
Topic: crawldata
Live Web Proxy Crawls
liveweb-20170830061935
web
eye 2.9M
favorite 0
comment 0