ArchiveBot: The Archive Team Crowdsourced Crawler
ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites).
To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need...
Filters
37,773
Results
Year Published range filter
-
Part Of
Media Type filters
Year filters
Subject filters
Collection filters
Creator filters
Language filters
Description
ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites).To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel operator permissions in order to issue archiving jobs. The dashboard shows the sites being downloaded currently.
There is a dashboard running for the archivebot process at http://www.archivebot.com.
ArchiveBot's source code can be found at https://github.com/ArchiveTeam/ArchiveBot.
Activity
Collection Info
- Addeddate
- 2014-04-08 17:05:17
- Collection
- Identifier
- archivebot
- Mediatype
- collection
- Num-recent-reviews
- 5
- Num-top-dl
- 5
- Publicdate
- 2014-04-08 17:05:17
- Storage_size
- 3.8 PB (in 4,515,493 files)
- Subject
- Title
- ArchiveBot: The Archive Team Crowdsourced Crawler
Web Archive Summary
Additional contributors
Views
Total Views: 3,639,297,638
Jan
2017
2017
Jan
2018
2018
Jan
2019
2019
Jan
2020
2020
Jan
2021
2021
Jan
2022
2022
Jan
2023
2023
Jan
2024
2024
0
20M
40M
60M
80M
100M
|
by people |
|
by robots |
Items
Total Items: 37,716
Jan
2015
2015
Jan
2016
2016
Jan
2016
2016
Jan
2018
2018
Jan
2019
2019
Jan
2020
2020
Jan
2020
2020
Jan
2022
2022
Jan
2023
2023
Jan
2024
2024
0
10K
20K
30K
40K
Region | Country | Views |
---|---|---|
Ontario | Canada | 27,308 |
Wellington | New Zealand | 5,518 |
Hunan | China | 3,044 |
California | United States of America | 2,171 |
Colorado | United States of America | 449 |
Unknown | United States of America | 296 |
Virginia | United States of America | 184 |
Unknown | 153 | |
Texas | United States of America | 114 |
Washington | United States of America | 88 |